Sep 17 18:02:53.133906 ip-10-0-138-138 systemd[1]: Starting Kubernetes Kubelet... Sep 17 18:02:53.517036 ip-10-0-138-138 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 18:02:53.517036 ip-10-0-138-138 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 18:02:53.517036 ip-10-0-138-138 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 18:02:53.517036 ip-10-0-138-138 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 18:02:53.517036 ip-10-0-138-138 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 18:02:53.518701 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.518635 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 18:02:53.522918 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522904 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:02:53.522918 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522918 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522922 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522926 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522929 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522931 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522934 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522936 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522939 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522942 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522944 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522947 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522951 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522955 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522961 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522964 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522967 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522969 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522972 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522975 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522977 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:02:53.522981 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522979 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522982 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522985 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522987 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522990 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522993 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.522996 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523000 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523005 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523008 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523011 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523015 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523018 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523021 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523024 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523026 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523029 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523032 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523034 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:02:53.523484 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523037 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523039 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523042 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523045 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523047 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523050 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523052 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523055 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523058 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523060 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523062 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523065 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523067 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523069 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523072 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523075 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523077 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523080 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523083 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523085 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:02:53.523957 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523088 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523090 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523093 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523095 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523098 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523100 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523103 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523105 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523108 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523110 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523113 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523115 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523117 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523120 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523122 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523125 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523127 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523130 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523132 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:02:53.524426 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523134 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523137 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523140 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523142 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523144 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523147 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523149 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523151 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523156 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523159 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523161 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523164 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523166 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523169 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523171 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523174 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523177 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523179 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523182 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:02:53.524886 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.523184 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:02:53.525527 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525516 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:02:53.525527 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525528 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525531 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525534 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525538 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525541 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525544 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525547 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525549 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525552 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525555 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525557 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525560 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525562 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525565 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525568 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525571 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525573 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525576 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525578 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:02:53.525579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525581 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525584 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525587 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525590 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525592 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525595 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525597 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525600 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525602 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525604 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525607 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525609 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525612 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525614 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525616 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525631 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525634 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525636 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525639 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525642 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:02:53.526099 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525644 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525647 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525649 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525652 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525654 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525657 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525661 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525665 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525668 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525671 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525673 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525677 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525680 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525683 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525688 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525691 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525694 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525697 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525707 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:02:53.526581 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525712 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525716 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525719 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525722 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525725 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525727 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525730 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525733 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525735 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525738 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525740 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525744 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525747 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525749 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525752 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525754 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525757 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525759 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525761 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525764 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:02:53.527066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525766 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525769 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525772 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525776 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525779 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525782 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525784 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525789 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525792 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525794 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525797 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525799 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525802 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525804 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525807 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525809 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525812 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525814 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525817 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:02:53.527542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.525819 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526167 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526173 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526176 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526180 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526183 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526186 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526189 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526192 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526194 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526197 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526200 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526203 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526206 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526209 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526212 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526215 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526217 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526220 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:02:53.528010 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526222 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526226 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526228 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526231 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526233 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526236 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526239 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526241 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526244 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526246 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526248 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526251 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526254 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526256 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526258 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526261 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526263 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526265 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526268 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:02:53.528467 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526270 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526273 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526275 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526277 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526281 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526284 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526288 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526290 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526293 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526295 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526298 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526331 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526334 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526336 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526339 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526342 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526345 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526348 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526350 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526353 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:02:53.528929 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526355 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526358 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526360 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526363 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526365 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526368 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526370 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526373 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526375 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526377 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526379 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526382 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526384 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526387 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526389 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526392 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526394 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526397 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526399 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526403 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:02:53.529407 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526405 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526407 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526410 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526413 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526417 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526419 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526422 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526425 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526427 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526430 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526432 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526434 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526437 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526441 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526445 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526447 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526450 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526452 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526454 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526457 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:02:53.529896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526459 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.526462 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526532 2564 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526581 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526589 2564 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526593 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526597 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526600 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526631 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526637 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526641 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526644 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526647 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526650 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526653 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526657 2564 flags.go:64] FLAG: --cgroup-root="" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526661 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526664 2564 flags.go:64] FLAG: --client-ca-file="" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526668 2564 flags.go:64] FLAG: --cloud-provider="external" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526670 2564 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526674 2564 flags.go:64] FLAG: --cluster-domain="" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526677 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526680 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526683 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 18:02:53.530369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526686 2564 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526689 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526692 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526695 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526698 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526701 2564 flags.go:64] FLAG: --contention-profiling="false" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526704 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526707 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526710 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526713 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526716 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526719 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526722 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526725 2564 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526727 2564 flags.go:64] FLAG: --enable-server="true" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526730 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526734 2564 flags.go:64] FLAG: --event-burst="100" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526737 2564 flags.go:64] FLAG: --event-qps="50" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526740 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526743 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526746 2564 flags.go:64] FLAG: --eviction-hard="" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526749 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526752 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526755 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526759 2564 flags.go:64] FLAG: --eviction-soft="" Sep 17 18:02:53.531007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526762 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526766 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526769 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526771 2564 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526774 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526777 2564 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526779 2564 flags.go:64] FLAG: --feature-gates="" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526783 2564 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526786 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526789 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526792 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526795 2564 flags.go:64] FLAG: --healthz-port="10248" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526798 2564 flags.go:64] FLAG: --help="false" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526800 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-138-138.ec2.internal" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526803 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526806 2564 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526809 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526812 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526815 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526818 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526821 2564 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526823 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526826 2564 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 18:02:53.531643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526829 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526832 2564 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526835 2564 flags.go:64] FLAG: --kube-reserved="" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526837 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526840 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526843 2564 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526846 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526849 2564 flags.go:64] FLAG: --lock-file="" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526853 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526856 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526860 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526865 2564 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526868 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526870 2564 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526873 2564 flags.go:64] FLAG: --logging-format="text" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526876 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526879 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526882 2564 flags.go:64] FLAG: --manifest-url="" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526884 2564 flags.go:64] FLAG: --manifest-url-header="" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526888 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526891 2564 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526895 2564 flags.go:64] FLAG: --max-pods="110" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526898 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526901 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526904 2564 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 18:02:53.532221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526906 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526909 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526912 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526915 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526922 2564 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526925 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526927 2564 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526930 2564 flags.go:64] FLAG: --pod-cidr="" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526933 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526938 2564 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526941 2564 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526944 2564 flags.go:64] FLAG: --pods-per-core="0" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526947 2564 flags.go:64] FLAG: --port="10250" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526950 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526953 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05286f7e7bd4c59c1" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526956 2564 flags.go:64] FLAG: --qos-reserved="" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526960 2564 flags.go:64] FLAG: --read-only-port="10255" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526965 2564 flags.go:64] FLAG: --register-node="true" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526968 2564 flags.go:64] FLAG: --register-with-taints="" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526971 2564 flags.go:64] FLAG: --registry-burst="10" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526974 2564 flags.go:64] FLAG: --registry-qps="5" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526977 2564 flags.go:64] FLAG: --reserved-cpus="" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526979 2564 flags.go:64] FLAG: --reserved-memory="" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526983 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526986 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 18:02:53.532871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526989 2564 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526992 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526995 2564 flags.go:64] FLAG: --runonce="false" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.526997 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527000 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527003 2564 flags.go:64] FLAG: --seccomp-default="false" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527006 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527009 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527012 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527014 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527018 2564 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527021 2564 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527023 2564 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527026 2564 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527029 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527032 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527035 2564 flags.go:64] FLAG: --system-cgroups="" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527037 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527043 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527045 2564 flags.go:64] FLAG: --tls-cert-file="" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527048 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527051 2564 flags.go:64] FLAG: --tls-min-version="" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527054 2564 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527057 2564 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527061 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 18:02:53.533466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527065 2564 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527067 2564 flags.go:64] FLAG: --v="2" Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527071 2564 flags.go:64] FLAG: --version="false" Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527075 2564 flags.go:64] FLAG: --vmodule="" Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527079 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527082 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527196 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527200 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527203 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527206 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527208 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527211 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527213 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527216 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527218 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527221 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527224 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527227 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527229 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527232 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527235 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:02:53.534353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527237 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527239 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527242 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527244 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527247 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527249 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527252 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527255 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527257 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527261 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527265 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527267 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527274 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527277 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527279 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527282 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527284 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527287 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527289 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527292 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:02:53.535018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527294 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527297 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527300 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527302 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527304 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527307 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527309 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527312 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527314 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527317 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527319 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527322 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527324 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527327 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527331 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527334 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527337 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527340 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527345 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:02:53.535544 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527348 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527350 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527354 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527360 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527362 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527365 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527368 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527370 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527373 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527375 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527378 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527381 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527383 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527386 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527388 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527391 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527393 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527395 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527398 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527401 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:02:53.536028 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527403 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527406 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527408 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527411 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527413 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527417 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527420 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527423 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527426 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527428 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527431 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527434 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527436 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527439 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527442 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527446 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527448 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527451 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527453 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:02:53.536514 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527456 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527459 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527461 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527464 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527466 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.527469 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.527473 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.535546 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.535568 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.536377 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.536385 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536442 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536447 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536450 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536454 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:02:53.536999 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536457 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536459 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536462 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536465 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536468 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536471 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536473 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536476 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536479 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536483 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536485 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536488 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536490 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536493 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536496 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536498 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536501 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536503 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536506 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:02:53.537368 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536508 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536511 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536513 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536516 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536518 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536521 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536524 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536527 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536530 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536532 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536535 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536537 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536540 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536543 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536545 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536548 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536550 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536553 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536556 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536559 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:02:53.537876 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536561 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536564 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536567 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536570 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536572 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536575 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536577 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536580 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536583 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536585 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536587 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536590 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536593 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536595 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536598 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536600 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536602 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536605 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536608 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536611 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:02:53.538353 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536613 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536617 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536636 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536639 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536642 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536644 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536647 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536650 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536654 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536657 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536660 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536662 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536665 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536669 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536672 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536675 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536678 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536681 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536684 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:02:53.538889 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536687 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536689 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536692 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536695 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536698 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536701 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536703 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536706 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536709 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536711 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536715 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536718 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536721 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536723 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536726 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536728 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:02:53.539355 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536731 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.536736 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536844 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536849 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536853 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536857 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536859 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536862 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536865 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536868 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536870 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536873 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536875 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536878 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536881 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:02:53.539812 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536883 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536886 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536888 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536891 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536894 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536896 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536898 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536901 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536903 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536906 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536908 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536911 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536914 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536916 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536919 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536921 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536923 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536926 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536928 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:02:53.540182 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536931 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536933 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536936 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536938 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536941 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536943 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536945 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536948 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536950 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536952 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536956 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536959 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536962 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536965 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536968 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536971 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536973 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536975 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536978 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:02:53.540663 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536980 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536983 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536985 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536988 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536990 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536993 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536996 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.536998 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537001 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537003 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537006 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537008 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537011 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537014 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537016 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537018 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537021 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537023 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537026 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537028 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:02:53.541118 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537031 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537033 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537036 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537038 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537040 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537043 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537045 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537048 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537051 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537053 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537056 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537058 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537061 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537063 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537067 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537070 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537073 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537075 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537078 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:02:53.541579 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537081 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537084 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537086 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537088 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537091 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537093 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537096 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537098 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:53.537100 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.537105 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.537205 2564 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.539274 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.540206 2564 server.go:1019] "Starting client certificate rotation" Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.540304 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 18:02:53.542168 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.540333 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 18:02:53.542742 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.542729 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 18:02:53.543787 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.543771 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 18:02:53.555116 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.555098 2564 log.go:25] "Validated CRI v1 runtime API" Sep 17 18:02:53.556715 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.556700 2564 log.go:25] "Validated CRI v1 image API" Sep 17 18:02:53.558300 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.558289 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 18:02:53.561457 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.561434 2564 fs.go:135] Filesystem UUIDs: map[3a1d37e2-bb8c-4eb6-88e8-916c74ac5bab:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ff48878f-eb1d-4c95-a85b-81aa34bf8f4d:/dev/nvme0n1p4] Sep 17 18:02:53.561530 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.561456 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 18:02:53.565158 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.565045 2564 manager.go:216] Machine: {Timestamp:2026-09-17 18:02:53.564557727 +0000 UTC m=+0.320650186 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100291 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec291f6ab4edc71ed1ad210510c998b4 SystemUUID:ec291f6a-b4ed-c71e-d1ad-210510c998b4 BootID:215659c2-d5e6-4f20-9032-ab587e3aa447 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:d9:37:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:d9:37:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:90:09:87:7d:6c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 18:02:53.565158 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.565148 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 18:02:53.565294 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.565210 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 18:02:53.568399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.568203 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 18:02:53.568533 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.568405 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-138.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 18:02:53.569806 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.569795 2564 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 18:02:53.569846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.569808 2564 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 18:02:53.569846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.569815 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 18:02:53.570684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.570674 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 18:02:53.572046 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.572037 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 18:02:53.574895 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.574883 2564 state_mem.go:36] "Initialized new in-memory state store" Sep 17 18:02:53.574999 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.574990 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 18:02:53.575030 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.575022 2564 kubelet.go:486] "Attempting to sync node with API server" Sep 17 18:02:53.575060 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.575031 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 18:02:53.575060 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.575042 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 18:02:53.575060 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.575051 2564 kubelet.go:398] "Adding apiserver pod source" Sep 17 18:02:53.575060 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.575059 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 18:02:53.575390 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.575377 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 18:02:53.575426 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.575395 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 18:02:53.577396 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.577379 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 17 18:02:53.579319 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.579302 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 18:02:53.579886 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.579874 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 18:02:53.579934 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.579904 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 18:02:53.580983 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.580970 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 18:02:53.581038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.580986 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 18:02:53.581038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.580992 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 18:02:53.581038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.580997 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 18:02:53.581038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.581005 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 18:02:53.581038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.581010 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 18:02:53.581038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.581026 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 18:02:53.581038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.581034 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 18:02:53.581038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.581041 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 18:02:53.581245 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.581046 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 18:02:53.581245 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.581054 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 18:02:53.581245 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.581059 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 18:02:53.581743 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.581731 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 18:02:53.581782 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.581744 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 18:02:53.585222 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.585208 2564 server.go:1268] "Started kubelet" Sep 17 18:02:53.585308 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.585285 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 18:02:53.585405 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.585381 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 18:02:53.585454 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.585425 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 18:02:53.585975 ip-10-0-138-138 systemd[1]: Started Kubernetes Kubelet. Sep 17 18:02:53.586587 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.586566 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 18:02:53.587321 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.587307 2564 server.go:310] "Adding debug handlers to kubelet server" Sep 17 18:02:53.588116 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.588095 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-138.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 18:02:53.588209 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.588180 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 18:02:53.588296 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.588279 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 18:02:53.589490 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.589466 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 18:02:53.593476 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.593457 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 18:02:53.593961 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.593941 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 18:02:53.594144 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.594089 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Sep 17 18:02:53.595004 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.594860 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 18:02:53.595109 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.595011 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 18:02:53.595109 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.594979 2564 factory.go:55] Registering systemd factory Sep 17 18:02:53.595109 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.595049 2564 factory.go:223] Registration of the systemd container factory successfully Sep 17 18:02:53.595109 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.594887 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 18:02:53.595311 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.595144 2564 reconstruct.go:97] "Volume reconstruction finished" Sep 17 18:02:53.595311 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.595154 2564 reconciler.go:29] "Reconciler: start to sync state" Sep 17 18:02:53.595311 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.595279 2564 factory.go:153] Registering CRI-O factory Sep 17 18:02:53.595311 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.595291 2564 factory.go:223] Registration of the crio container factory successfully Sep 17 18:02:53.595486 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.595344 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 18:02:53.595486 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.595370 2564 factory.go:103] Registering Raw factory Sep 17 18:02:53.595486 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.595383 2564 manager.go:1196] Started watching for new ooms in manager Sep 17 18:02:53.595845 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.595830 2564 manager.go:349] Starting recovery of all containers Sep 17 18:02:53.604796 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.598768 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 18:02:53.604796 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.598858 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 18:02:53.605610 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.601824 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-138.ec2.internal.18d62db8c85dd58b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-138.ec2.internal,UID:ip-10-0-138-138.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-138.ec2.internal,},FirstTimestamp:2026-09-17 18:02:53.585175947 +0000 UTC m=+0.341268410,LastTimestamp:2026-09-17 18:02:53.585175947 +0000 UTC m=+0.341268410,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-138.ec2.internal,}" Sep 17 18:02:53.608740 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.608700 2564 manager.go:354] Recovery completed Sep 17 18:02:53.612041 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.612012 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w7wzl" Sep 17 18:02:53.613978 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.613967 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:02:53.617016 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.617001 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:02:53.617063 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.617029 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:02:53.617063 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.617040 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:02:53.617571 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.617555 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 18:02:53.617571 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.617567 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:02:53.617571 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.617570 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 18:02:53.617755 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.617588 2564 state_mem.go:36] "Initialized new in-memory state store" Sep 17 18:02:53.618833 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.618822 2564 policy_none.go:49] "None policy: Start" Sep 17 18:02:53.618868 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.618838 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 18:02:53.618868 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.618847 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 18:02:53.619117 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.619092 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w7wzl" Sep 17 18:02:53.619294 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.619275 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:02:53.619378 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.619305 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:02:53.619378 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.619315 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:02:53.619472 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.619332 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-138.ec2.internal.18d62db8ca43ad5e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-138.ec2.internal,UID:ip-10-0-138-138.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-138.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-138.ec2.internal,},FirstTimestamp:2026-09-17 18:02:53.617016158 +0000 UTC m=+0.373108617,LastTimestamp:2026-09-17 18:02:53.617016158 +0000 UTC m=+0.373108617,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-138.ec2.internal,}" Sep 17 18:02:53.620530 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.620517 2564 policy_none.go:47] "Start" Sep 17 18:02:53.655236 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.655222 2564 manager.go:337] "Starting Device Plugin manager" Sep 17 18:02:53.664935 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.655266 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 18:02:53.664935 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.655275 2564 server.go:87] "Starting device plugin registration server" Sep 17 18:02:53.664935 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.655551 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 18:02:53.664935 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.655564 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 18:02:53.664935 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.655685 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 18:02:53.664935 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.655776 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 18:02:53.664935 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.655793 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 18:02:53.664935 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.656122 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 18:02:53.664935 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.656179 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-138.ec2.internal\" not found" Sep 17 18:02:53.696991 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.696973 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 18:02:53.698258 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.698238 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 18:02:53.698329 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.698280 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 18:02:53.698329 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.698300 2564 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 18:02:53.698412 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.698332 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 18:02:53.701060 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.701043 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:02:53.755905 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.755885 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:02:53.756691 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.756677 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:02:53.756758 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.756701 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:02:53.756758 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.756711 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:02:53.756758 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.756731 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-138.ec2.internal" Sep 17 18:02:53.765770 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.765753 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-138.ec2.internal" Sep 17 18:02:53.765819 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.765772 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-138.ec2.internal\": node \"ip-10-0-138-138.ec2.internal\" not found" Sep 17 18:02:53.799108 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.799054 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal"] Sep 17 18:02:53.799163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.799108 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:02:53.799872 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.799859 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:02:53.799939 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.799884 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:02:53.799939 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.799899 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:02:53.801096 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.801080 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Sep 17 18:02:53.801185 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.801175 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:02:53.801319 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.801305 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Sep 17 18:02:53.801351 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.801335 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:02:53.801857 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.801845 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:02:53.801919 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.801865 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:02:53.801919 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.801886 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:02:53.801919 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.801896 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:02:53.802020 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.801867 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:02:53.802020 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.801952 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:02:53.803267 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.803246 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" Sep 17 18:02:53.803359 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.803279 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:02:53.803996 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.803977 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:02:53.804079 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.804005 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:02:53.804079 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.804019 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:02:53.827162 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.827145 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-138.ec2.internal\" not found" node="ip-10-0-138-138.ec2.internal" Sep 17 18:02:53.831753 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.831739 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-138.ec2.internal\" not found" node="ip-10-0-138-138.ec2.internal" Sep 17 18:02:53.902094 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:53.902077 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Sep 17 18:02:53.996234 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.996207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf72d80b110564496492d5b5b6546081-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal\" (UID: \"cf72d80b110564496492d5b5b6546081\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Sep 17 18:02:53.996321 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.996253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf72d80b110564496492d5b5b6546081-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal\" (UID: \"cf72d80b110564496492d5b5b6546081\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Sep 17 18:02:53.996321 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:53.996309 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d18622155b9da2257267bc123b9432af-config\") pod \"kube-apiserver-proxy-ip-10-0-138-138.ec2.internal\" (UID: \"d18622155b9da2257267bc123b9432af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" Sep 17 18:02:54.002184 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:54.002169 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Sep 17 18:02:54.097110 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.097062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf72d80b110564496492d5b5b6546081-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal\" (UID: \"cf72d80b110564496492d5b5b6546081\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Sep 17 18:02:54.097110 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.097111 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf72d80b110564496492d5b5b6546081-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal\" (UID: \"cf72d80b110564496492d5b5b6546081\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Sep 17 18:02:54.097193 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.097149 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf72d80b110564496492d5b5b6546081-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal\" (UID: \"cf72d80b110564496492d5b5b6546081\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Sep 17 18:02:54.097193 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.097173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d18622155b9da2257267bc123b9432af-config\") pod \"kube-apiserver-proxy-ip-10-0-138-138.ec2.internal\" (UID: \"d18622155b9da2257267bc123b9432af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" Sep 17 18:02:54.097256 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.097220 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d18622155b9da2257267bc123b9432af-config\") pod \"kube-apiserver-proxy-ip-10-0-138-138.ec2.internal\" (UID: \"d18622155b9da2257267bc123b9432af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" Sep 17 18:02:54.097256 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.097227 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf72d80b110564496492d5b5b6546081-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal\" (UID: \"cf72d80b110564496492d5b5b6546081\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Sep 17 18:02:54.103184 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:54.103166 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Sep 17 18:02:54.131320 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.131305 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Sep 17 18:02:54.136574 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.136548 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" Sep 17 18:02:54.203669 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:54.203647 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Sep 17 18:02:54.304222 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:54.304201 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Sep 17 18:02:54.404745 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:54.404704 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Sep 17 18:02:54.505323 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:54.505305 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Sep 17 18:02:54.540473 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.540449 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 18:02:54.540872 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.540616 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 18:02:54.605358 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:54.605341 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Sep 17 18:02:54.607558 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.607543 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:02:54.623665 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.623612 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:57:55 +0000 UTC" deadline="2028-06-12 04:30:49.395062774 +0000 UTC" Sep 17 18:02:54.623665 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.623665 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15202h27m54.771402151s" Sep 17 18:02:54.706319 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:54.706302 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Sep 17 18:02:54.713545 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:54.713522 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf72d80b110564496492d5b5b6546081.slice/crio-3a0c1af52c86efb177de28b40658237e3429cf91be1c3c2b5bd265b6d64b86c6 WatchSource:0}: Error finding container 3a0c1af52c86efb177de28b40658237e3429cf91be1c3c2b5bd265b6d64b86c6: Status 404 returned error can't find the container with id 3a0c1af52c86efb177de28b40658237e3429cf91be1c3c2b5bd265b6d64b86c6 Sep 17 18:02:54.717110 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.717096 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:02:54.736068 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.736048 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:02:54.795161 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.795143 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Sep 17 18:02:54.801386 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:54.801368 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18622155b9da2257267bc123b9432af.slice/crio-29302f722d81c4e0c4822a01742fcb204f69befc16fbe988d80ba6900d87192a WatchSource:0}: Error finding container 29302f722d81c4e0c4822a01742fcb204f69befc16fbe988d80ba6900d87192a: Status 404 returned error can't find the container with id 29302f722d81c4e0c4822a01742fcb204f69befc16fbe988d80ba6900d87192a Sep 17 18:02:54.809279 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.809263 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 18:02:54.811140 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.811127 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" Sep 17 18:02:54.821509 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.821495 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 18:02:54.834237 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:54.834218 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:02:55.514899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.514872 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:02:55.576385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.576357 2564 apiserver.go:52] "Watching apiserver" Sep 17 18:02:55.586219 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.586194 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 18:02:55.588362 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.588332 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-mt8dc","openshift-ovn-kubernetes/ovnkube-node-qx27j","kube-system/konnectivity-agent-f87tt","kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal","openshift-dns/node-resolver-h9q2n","openshift-multus/multus-l529v","openshift-network-diagnostics/network-check-target-b4sz9","openshift-network-operator/iptables-alerter-5wnp7","kube-system/global-pull-secret-syncer-l2fvr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk","openshift-cluster-node-tuning-operator/tuned-k9dvm","openshift-image-registry/node-ca-8hr8c","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal","openshift-multus/multus-additional-cni-plugins-q7k8g"] Sep 17 18:02:55.590129 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.590108 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.591697 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.591677 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.593811 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.593789 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 18:02:55.594142 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.594121 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mnbk2\"" Sep 17 18:02:55.594389 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.594371 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:02:55.594472 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.594400 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 18:02:55.594617 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.594602 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 18:02:55.594715 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.594640 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f87tt" Sep 17 18:02:55.595193 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.595176 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lk6pt\"" Sep 17 18:02:55.595275 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.595181 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 18:02:55.595350 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.595321 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 18:02:55.596496 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.596477 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.598319 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.598088 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h9q2n" Sep 17 18:02:55.601108 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.601087 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l529v" Sep 17 18:02:55.601597 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.601577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:02:55.601708 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:55.601665 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:02:55.603814 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.603794 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5wnp7" Sep 17 18:02:55.604303 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-device-dir\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.604374 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604309 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-modprobe-d\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.604374 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-host\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.604374 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604338 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2tjz\" (UniqueName: \"kubernetes.io/projected/b27232ef-6a25-479f-b137-8ae12d97efd6-kube-api-access-q2tjz\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.604374 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.604541 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-etc-selinux\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.604541 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604404 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-sys-fs\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.604541 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-run\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.604541 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7gk6\" (UniqueName: \"kubernetes.io/projected/36401200-8357-4c9b-9f41-7a13eb41f824-kube-api-access-r7gk6\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.604541 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-sys\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.604541 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604474 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b27232ef-6a25-479f-b137-8ae12d97efd6-tmp\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.604541 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604491 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e0a74f-1624-4aaa-b102-d15049b61cca-system-cni-dir\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.604541 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40e0a74f-1624-4aaa-b102-d15049b61cca-cnibin\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.604541 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-registration-dir\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604535 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21e09d04-55d0-4833-8f9c-ec78ea8beffc-agent-certs\") pod \"konnectivity-agent-f87tt\" (UID: \"21e09d04-55d0-4833-8f9c-ec78ea8beffc\") " pod="kube-system/konnectivity-agent-f87tt" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40e0a74f-1624-4aaa-b102-d15049b61cca-cni-binary-copy\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21e09d04-55d0-4833-8f9c-ec78ea8beffc-konnectivity-ca\") pod \"konnectivity-agent-f87tt\" (UID: \"21e09d04-55d0-4833-8f9c-ec78ea8beffc\") " pod="kube-system/konnectivity-agent-f87tt" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-sysctl-d\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-sysctl-conf\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-lib-modules\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-tuned\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb7kj\" (UniqueName: \"kubernetes.io/projected/40e0a74f-1624-4aaa-b102-d15049b61cca-kube-api-access-mb7kj\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-socket-dir\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-sysconfig\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-kubernetes\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604841 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-systemd\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40e0a74f-1624-4aaa-b102-d15049b61cca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.604877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-var-lib-kubelet\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.605549 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40e0a74f-1624-4aaa-b102-d15049b61cca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.605549 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.604963 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40e0a74f-1624-4aaa-b102-d15049b61cca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.605549 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.605246 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.606816 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.606796 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:02:55.606904 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:55.606878 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:02:55.608830 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.608809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:02:55.608915 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:55.608868 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:02:55.610163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.610146 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8hr8c" Sep 17 18:02:55.611699 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.611667 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 18:02:55.612013 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.611994 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 18:02:55.612236 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.612220 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 18:02:55.612304 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.612287 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 18:02:55.612426 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.612400 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 18:02:55.612494 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.612446 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 18:02:55.612544 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.612222 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 18:02:55.612705 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.612687 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 18:02:55.613584 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.612877 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 18:02:55.613584 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.613235 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 18:02:55.613584 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.613484 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 18:02:55.613789 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.613616 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 18:02:55.613839 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.613819 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 18:02:55.613945 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.613929 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 18:02:55.614171 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.614156 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 18:02:55.614764 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.614738 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9stnd\"" Sep 17 18:02:55.614903 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.614886 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 18:02:55.615006 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.614992 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:02:55.615319 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.615302 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2nzwr\"" Sep 17 18:02:55.615417 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.615402 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k4c76\"" Sep 17 18:02:55.615479 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.615417 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ccbh2\"" Sep 17 18:02:55.615721 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.615702 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9tgtr\"" Sep 17 18:02:55.615819 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.615718 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 18:02:55.615819 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.615772 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wpp9z\"" Sep 17 18:02:55.616218 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.615944 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 18:02:55.616218 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.615974 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 18:02:55.616218 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.615990 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 18:02:55.616218 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.615321 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 18:02:55.616405 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.616384 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r5f56\"" Sep 17 18:02:55.623337 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.623316 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 18:02:55.662826 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.662805 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rbxv2" Sep 17 18:02:55.677877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.677858 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rbxv2" Sep 17 18:02:55.696884 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.696863 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 18:02:55.703237 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.703194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" event={"ID":"d18622155b9da2257267bc123b9432af","Type":"ContainerStarted","Data":"29302f722d81c4e0c4822a01742fcb204f69befc16fbe988d80ba6900d87192a"} Sep 17 18:02:55.704796 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.704769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" event={"ID":"cf72d80b110564496492d5b5b6546081","Type":"ContainerStarted","Data":"3a0c1af52c86efb177de28b40658237e3429cf91be1c3c2b5bd265b6d64b86c6"} Sep 17 18:02:55.706027 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-registration-dir\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.706109 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40e0a74f-1624-4aaa-b102-d15049b61cca-cni-binary-copy\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.706109 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706076 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-system-cni-dir\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.706109 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-var-lib-cni-bin\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.706225 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lv8j\" (UniqueName: \"kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j\") pod \"network-check-target-b4sz9\" (UID: \"e4f805c2-e759-4c84-88c4-5d6e5e34de8f\") " pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:02:55.706225 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-run-netns\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.706225 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-sysctl-d\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.706225 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706153 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-var-lib-kubelet\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.706225 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-node-log\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.706225 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-socket-dir\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.706225 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-sysconfig\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.706535 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-kubernetes\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.706535 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706385 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-sysctl-d\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.706535 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706402 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-socket-dir\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.706535 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706455 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-sysconfig\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.706535 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706469 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-multus-cni-dir\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.706535 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0704f62c-880a-451b-851f-731517d14596-dbus\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:02:55.706535 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80528350-3ca9-493f-94f7-39ddbf06ce11-host-slash\") pod \"iptables-alerter-5wnp7\" (UID: \"80528350-3ca9-493f-94f7-39ddbf06ce11\") " pod="openshift-network-operator/iptables-alerter-5wnp7" Sep 17 18:02:55.706831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8fj8\" (UniqueName: \"kubernetes.io/projected/18d9adda-d73b-4456-83d8-360539054c24-kube-api-access-x8fj8\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:02:55.706831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b73423cd-ef53-471d-8b77-c72793a394b8-hosts-file\") pod \"node-resolver-h9q2n\" (UID: \"b73423cd-ef53-471d-8b77-c72793a394b8\") " pod="openshift-dns/node-resolver-h9q2n" Sep 17 18:02:55.706831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-var-lib-kubelet\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.706831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706651 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-registration-dir\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.706831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73423cd-ef53-471d-8b77-c72793a394b8-tmp-dir\") pod \"node-resolver-h9q2n\" (UID: \"b73423cd-ef53-471d-8b77-c72793a394b8\") " pod="openshift-dns/node-resolver-h9q2n" Sep 17 18:02:55.706831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706686 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-var-lib-kubelet\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.706831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706691 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.706831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-ovnkube-config\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.706831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706764 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40e0a74f-1624-4aaa-b102-d15049b61cca-cni-binary-copy\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.706831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706774 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-device-dir\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.706831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706763 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-kubernetes\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.706831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-host\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.706831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2tjz\" (UniqueName: \"kubernetes.io/projected/b27232ef-6a25-479f-b137-8ae12d97efd6-kube-api-access-q2tjz\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d1b7fa0-4d80-496a-a4ce-02a720060731-cni-binary-copy\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-cni-bin\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706875 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-device-dir\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706890 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-host\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706930 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-sys-fs\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d1b7fa0-4d80-496a-a4ce-02a720060731-multus-daemon-config\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-kubelet\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.706985 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-var-lib-openvswitch\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-sys-fs\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-run-openvswitch\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7gk6\" (UniqueName: \"kubernetes.io/projected/36401200-8357-4c9b-9f41-7a13eb41f824-kube-api-access-r7gk6\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-systemd-units\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707089 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-run-systemd\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707107 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-env-overrides\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.707343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40e0a74f-1624-4aaa-b102-d15049b61cca-cnibin\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707162 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jn4h\" (UniqueName: \"kubernetes.io/projected/6d1b7fa0-4d80-496a-a4ce-02a720060731-kube-api-access-4jn4h\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707192 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40e0a74f-1624-4aaa-b102-d15049b61cca-cnibin\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9fnn\" (UniqueName: \"kubernetes.io/projected/b73423cd-ef53-471d-8b77-c72793a394b8-kube-api-access-p9fnn\") pod \"node-resolver-h9q2n\" (UID: \"b73423cd-ef53-471d-8b77-c72793a394b8\") " pod="openshift-dns/node-resolver-h9q2n" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21e09d04-55d0-4833-8f9c-ec78ea8beffc-agent-certs\") pod \"konnectivity-agent-f87tt\" (UID: \"21e09d04-55d0-4833-8f9c-ec78ea8beffc\") " pod="kube-system/konnectivity-agent-f87tt" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707257 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-multus-socket-dir-parent\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707275 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2g4d\" (UniqueName: \"kubernetes.io/projected/80528350-3ca9-493f-94f7-39ddbf06ce11-kube-api-access-w2g4d\") pod \"iptables-alerter-5wnp7\" (UID: \"80528350-3ca9-493f-94f7-39ddbf06ce11\") " pod="openshift-network-operator/iptables-alerter-5wnp7" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-ovnkube-script-lib\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707402 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21e09d04-55d0-4833-8f9c-ec78ea8beffc-konnectivity-ca\") pod \"konnectivity-agent-f87tt\" (UID: \"21e09d04-55d0-4833-8f9c-ec78ea8beffc\") " pod="kube-system/konnectivity-agent-f87tt" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-sysctl-conf\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-lib-modules\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-tuned\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb7kj\" (UniqueName: \"kubernetes.io/projected/40e0a74f-1624-4aaa-b102-d15049b61cca-kube-api-access-mb7kj\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707521 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-run-k8s-cni-cncf-io\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707539 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-log-socket\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707557 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-cni-netd\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-systemd\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.707988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707576 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-sysctl-conf\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40e0a74f-1624-4aaa-b102-d15049b61cca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-systemd\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-run-multus-certs\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707707 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-lib-modules\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91d98768-2d1f-4172-a9c7-f782aa9056ac-serviceca\") pod \"node-ca-8hr8c\" (UID: \"91d98768-2d1f-4172-a9c7-f782aa9056ac\") " pod="openshift-image-registry/node-ca-8hr8c" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40e0a74f-1624-4aaa-b102-d15049b61cca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707849 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40e0a74f-1624-4aaa-b102-d15049b61cca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2vw\" (UniqueName: \"kubernetes.io/projected/91d98768-2d1f-4172-a9c7-f782aa9056ac-kube-api-access-6h2vw\") pod \"node-ca-8hr8c\" (UID: \"91d98768-2d1f-4172-a9c7-f782aa9056ac\") " pod="openshift-image-registry/node-ca-8hr8c" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707887 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21e09d04-55d0-4833-8f9c-ec78ea8beffc-konnectivity-ca\") pod \"konnectivity-agent-f87tt\" (UID: \"21e09d04-55d0-4833-8f9c-ec78ea8beffc\") " pod="kube-system/konnectivity-agent-f87tt" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707900 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-modprobe-d\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707922 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-hostroot\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707946 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-multus-conf-dir\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707926 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40e0a74f-1624-4aaa-b102-d15049b61cca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.708684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.707996 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0704f62c-880a-451b-851f-731517d14596-kubelet-config\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708002 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-modprobe-d\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/80528350-3ca9-493f-94f7-39ddbf06ce11-iptables-alerter-script\") pod \"iptables-alerter-5wnp7\" (UID: \"80528350-3ca9-493f-94f7-39ddbf06ce11\") " pod="openshift-network-operator/iptables-alerter-5wnp7" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-run-ovn\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-etc-selinux\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708071 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-run\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-cnibin\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708098 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40e0a74f-1624-4aaa-b102-d15049b61cca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-etc-openvswitch\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708138 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36401200-8357-4c9b-9f41-7a13eb41f824-etc-selinux\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708146 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-ovn-node-metrics-cert\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708163 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-run\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-sys\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b27232ef-6a25-479f-b137-8ae12d97efd6-tmp\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708204 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b27232ef-6a25-479f-b137-8ae12d97efd6-sys\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e0a74f-1624-4aaa-b102-d15049b61cca-system-cni-dir\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708234 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e0a74f-1624-4aaa-b102-d15049b61cca-system-cni-dir\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.709283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708234 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-run-netns\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.709982 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708260 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-var-lib-cni-multus\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.709982 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708284 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-slash\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.709982 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91d98768-2d1f-4172-a9c7-f782aa9056ac-host\") pod \"node-ca-8hr8c\" (UID: \"91d98768-2d1f-4172-a9c7-f782aa9056ac\") " pod="openshift-image-registry/node-ca-8hr8c" Sep 17 18:02:55.709982 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-etc-kubernetes\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:55.709982 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708358 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-run-ovn-kubernetes\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.709982 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708358 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40e0a74f-1624-4aaa-b102-d15049b61cca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.709982 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.708387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5h2x\" (UniqueName: \"kubernetes.io/projected/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-kube-api-access-n5h2x\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:55.711872 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.711832 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b27232ef-6a25-479f-b137-8ae12d97efd6-etc-tuned\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.712137 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.712120 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21e09d04-55d0-4833-8f9c-ec78ea8beffc-agent-certs\") pod \"konnectivity-agent-f87tt\" (UID: \"21e09d04-55d0-4833-8f9c-ec78ea8beffc\") " pod="kube-system/konnectivity-agent-f87tt" Sep 17 18:02:55.718170 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.718146 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2tjz\" (UniqueName: \"kubernetes.io/projected/b27232ef-6a25-479f-b137-8ae12d97efd6-kube-api-access-q2tjz\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.718886 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.718865 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b27232ef-6a25-479f-b137-8ae12d97efd6-tmp\") pod \"tuned-k9dvm\" (UID: \"b27232ef-6a25-479f-b137-8ae12d97efd6\") " pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:55.721011 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.720988 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb7kj\" (UniqueName: \"kubernetes.io/projected/40e0a74f-1624-4aaa-b102-d15049b61cca-kube-api-access-mb7kj\") pod \"multus-additional-cni-plugins-q7k8g\" (UID: \"40e0a74f-1624-4aaa-b102-d15049b61cca\") " pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:55.721114 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:55.721086 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7gk6\" (UniqueName: \"kubernetes.io/projected/36401200-8357-4c9b-9f41-7a13eb41f824-kube-api-access-r7gk6\") pod \"aws-ebs-csi-driver-node-wwnpk\" (UID: \"36401200-8357-4c9b-9f41-7a13eb41f824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:57.555758 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-multus-cni-dir\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.555758 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555749 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-multus-cni-dir\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.555947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0704f62c-880a-451b-851f-731517d14596-dbus\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:02:57.555947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80528350-3ca9-493f-94f7-39ddbf06ce11-host-slash\") pod \"iptables-alerter-5wnp7\" (UID: \"80528350-3ca9-493f-94f7-39ddbf06ce11\") " pod="openshift-network-operator/iptables-alerter-5wnp7" Sep 17 18:02:57.555947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555814 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0704f62c-880a-451b-851f-731517d14596-dbus\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:02:57.555947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555832 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8fj8\" (UniqueName: \"kubernetes.io/projected/18d9adda-d73b-4456-83d8-360539054c24-kube-api-access-x8fj8\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:02:57.555947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555853 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80528350-3ca9-493f-94f7-39ddbf06ce11-host-slash\") pod \"iptables-alerter-5wnp7\" (UID: \"80528350-3ca9-493f-94f7-39ddbf06ce11\") " pod="openshift-network-operator/iptables-alerter-5wnp7" Sep 17 18:02:57.555947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555856 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b73423cd-ef53-471d-8b77-c72793a394b8-hosts-file\") pod \"node-resolver-h9q2n\" (UID: \"b73423cd-ef53-471d-8b77-c72793a394b8\") " pod="openshift-dns/node-resolver-h9q2n" Sep 17 18:02:57.555947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73423cd-ef53-471d-8b77-c72793a394b8-tmp-dir\") pod \"node-resolver-h9q2n\" (UID: \"b73423cd-ef53-471d-8b77-c72793a394b8\") " pod="openshift-dns/node-resolver-h9q2n" Sep 17 18:02:57.555947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555895 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.555947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-ovnkube-config\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.555947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d1b7fa0-4d80-496a-a4ce-02a720060731-cni-binary-copy\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-cni-bin\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d1b7fa0-4d80-496a-a4ce-02a720060731-multus-daemon-config\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.555985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-kubelet\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-var-lib-openvswitch\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-run-openvswitch\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-systemd-units\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-run-systemd\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-env-overrides\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jn4h\" (UniqueName: \"kubernetes.io/projected/6d1b7fa0-4d80-496a-a4ce-02a720060731-kube-api-access-4jn4h\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9fnn\" (UniqueName: \"kubernetes.io/projected/b73423cd-ef53-471d-8b77-c72793a394b8-kube-api-access-p9fnn\") pod \"node-resolver-h9q2n\" (UID: \"b73423cd-ef53-471d-8b77-c72793a394b8\") " pod="openshift-dns/node-resolver-h9q2n" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-multus-socket-dir-parent\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556187 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2g4d\" (UniqueName: \"kubernetes.io/projected/80528350-3ca9-493f-94f7-39ddbf06ce11-kube-api-access-w2g4d\") pod \"iptables-alerter-5wnp7\" (UID: \"80528350-3ca9-493f-94f7-39ddbf06ce11\") " pod="openshift-network-operator/iptables-alerter-5wnp7" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-ovnkube-script-lib\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-run-k8s-cni-cncf-io\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556241 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-log-socket\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556255 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-cni-netd\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.556449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-run-multus-certs\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556289 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556310 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91d98768-2d1f-4172-a9c7-f782aa9056ac-serviceca\") pod \"node-ca-8hr8c\" (UID: \"91d98768-2d1f-4172-a9c7-f782aa9056ac\") " pod="openshift-image-registry/node-ca-8hr8c" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556340 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2vw\" (UniqueName: \"kubernetes.io/projected/91d98768-2d1f-4172-a9c7-f782aa9056ac-kube-api-access-6h2vw\") pod \"node-ca-8hr8c\" (UID: \"91d98768-2d1f-4172-a9c7-f782aa9056ac\") " pod="openshift-image-registry/node-ca-8hr8c" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-hostroot\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-multus-conf-dir\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0704f62c-880a-451b-851f-731517d14596-kubelet-config\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556411 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/80528350-3ca9-493f-94f7-39ddbf06ce11-iptables-alerter-script\") pod \"iptables-alerter-5wnp7\" (UID: \"80528350-3ca9-493f-94f7-39ddbf06ce11\") " pod="openshift-network-operator/iptables-alerter-5wnp7" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556418 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-ovnkube-config\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-run-ovn\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-cnibin\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-etc-openvswitch\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-ovn-node-metrics-cert\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556504 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-run-netns\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-var-lib-cni-multus\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556545 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-slash\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91d98768-2d1f-4172-a9c7-f782aa9056ac-host\") pod \"node-ca-8hr8c\" (UID: \"91d98768-2d1f-4172-a9c7-f782aa9056ac\") " pod="openshift-image-registry/node-ca-8hr8c" Sep 17 18:02:57.557330 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-etc-kubernetes\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-run-ovn-kubernetes\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556625 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5h2x\" (UniqueName: \"kubernetes.io/projected/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-kube-api-access-n5h2x\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-system-cni-dir\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-var-lib-cni-bin\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556667 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73423cd-ef53-471d-8b77-c72793a394b8-tmp-dir\") pod \"node-resolver-h9q2n\" (UID: \"b73423cd-ef53-471d-8b77-c72793a394b8\") " pod="openshift-dns/node-resolver-h9q2n" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lv8j\" (UniqueName: \"kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j\") pod \"network-check-target-b4sz9\" (UID: \"e4f805c2-e759-4c84-88c4-5d6e5e34de8f\") " pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-run-netns\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-var-lib-kubelet\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-node-log\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556782 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-node-log\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556972 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91d98768-2d1f-4172-a9c7-f782aa9056ac-serviceca\") pod \"node-ca-8hr8c\" (UID: \"91d98768-2d1f-4172-a9c7-f782aa9056ac\") " pod="openshift-image-registry/node-ca-8hr8c" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-hostroot\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557015 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-multus-conf-dir\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557069 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0704f62c-880a-451b-851f-731517d14596-kubelet-config\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557324 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d1b7fa0-4d80-496a-a4ce-02a720060731-cni-binary-copy\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557373 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-cni-bin\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.558163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557439 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/80528350-3ca9-493f-94f7-39ddbf06ce11-iptables-alerter-script\") pod \"iptables-alerter-5wnp7\" (UID: \"80528350-3ca9-493f-94f7-39ddbf06ce11\") " pod="openshift-network-operator/iptables-alerter-5wnp7" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557480 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-run-ovn\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557544 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-cnibin\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557586 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-run-ovn-kubernetes\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557594 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-etc-kubernetes\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557626 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-var-lib-cni-bin\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557643 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-etc-openvswitch\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557650 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-run-netns\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-run-netns\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.556470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b73423cd-ef53-471d-8b77-c72793a394b8-hosts-file\") pod \"node-resolver-h9q2n\" (UID: \"b73423cd-ef53-471d-8b77-c72793a394b8\") " pod="openshift-dns/node-resolver-h9q2n" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.557704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-var-lib-openvswitch\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.558006 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d1b7fa0-4d80-496a-a4ce-02a720060731-multus-daemon-config\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.558570 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-kubelet\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.558040 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91d98768-2d1f-4172-a9c7-f782aa9056ac-host\") pod \"node-ca-8hr8c\" (UID: \"91d98768-2d1f-4172-a9c7-f782aa9056ac\") " pod="openshift-image-registry/node-ca-8hr8c" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.558653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-var-lib-kubelet\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.558714 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-slash\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.558777 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-run-openvswitch\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.558897 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-var-lib-cni-multus\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.559065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.558916 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-system-cni-dir\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.559912 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.559091 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-multus-socket-dir-parent\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.559912 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.559149 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-host-cni-netd\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.559912 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.559443 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-ovnkube-script-lib\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.559912 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.559510 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-log-socket\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.559912 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:57.559639 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:02:57.559912 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.559653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-env-overrides\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.559912 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.559714 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-run-multus-certs\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.559912 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:57.559736 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret podName:0704f62c-880a-451b-851f-731517d14596 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:58.059699503 +0000 UTC m=+3.069886637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret") pod "global-pull-secret-syncer-l2fvr" (UID: "0704f62c-880a-451b-851f-731517d14596") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:02:57.560302 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:57.559968 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:02:57.560302 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:57.560023 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs podName:18d9adda-d73b-4456-83d8-360539054c24 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:58.06000299 +0000 UTC m=+3.070190123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs") pod "network-metrics-daemon-mt8dc" (UID: "18d9adda-d73b-4456-83d8-360539054c24") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:02:57.560302 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.560260 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d1b7fa0-4d80-496a-a4ce-02a720060731-host-run-k8s-cni-cncf-io\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.560448 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.560363 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-systemd-units\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.560985 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.560964 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-run-systemd\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.564549 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.563868 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-ovn-node-metrics-cert\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.570422 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.570402 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8fj8\" (UniqueName: \"kubernetes.io/projected/18d9adda-d73b-4456-83d8-360539054c24-kube-api-access-x8fj8\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:02:57.572656 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.572637 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9fnn\" (UniqueName: \"kubernetes.io/projected/b73423cd-ef53-471d-8b77-c72793a394b8-kube-api-access-p9fnn\") pod \"node-resolver-h9q2n\" (UID: \"b73423cd-ef53-471d-8b77-c72793a394b8\") " pod="openshift-dns/node-resolver-h9q2n" Sep 17 18:02:57.574929 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:57.574905 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:02:57.574929 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:57.574927 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:02:57.575086 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:57.574941 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7lv8j for pod openshift-network-diagnostics/network-check-target-b4sz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:02:57.575086 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:57.575008 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j podName:e4f805c2-e759-4c84-88c4-5d6e5e34de8f nodeName:}" failed. No retries permitted until 2026-09-17 18:02:58.074992506 +0000 UTC m=+3.085179635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7lv8j" (UniqueName: "kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j") pod "network-check-target-b4sz9" (UID: "e4f805c2-e759-4c84-88c4-5d6e5e34de8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:02:57.577047 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.576826 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2vw\" (UniqueName: \"kubernetes.io/projected/91d98768-2d1f-4172-a9c7-f782aa9056ac-kube-api-access-6h2vw\") pod \"node-ca-8hr8c\" (UID: \"91d98768-2d1f-4172-a9c7-f782aa9056ac\") " pod="openshift-image-registry/node-ca-8hr8c" Sep 17 18:02:57.578101 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.577933 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jn4h\" (UniqueName: \"kubernetes.io/projected/6d1b7fa0-4d80-496a-a4ce-02a720060731-kube-api-access-4jn4h\") pod \"multus-l529v\" (UID: \"6d1b7fa0-4d80-496a-a4ce-02a720060731\") " pod="openshift-multus/multus-l529v" Sep 17 18:02:57.578718 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.578693 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5h2x\" (UniqueName: \"kubernetes.io/projected/9c2e6ff5-15fc-4af0-8095-8c7627bc7110-kube-api-access-n5h2x\") pod \"ovnkube-node-qx27j\" (UID: \"9c2e6ff5-15fc-4af0-8095-8c7627bc7110\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.579763 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.579724 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2g4d\" (UniqueName: \"kubernetes.io/projected/80528350-3ca9-493f-94f7-39ddbf06ce11-kube-api-access-w2g4d\") pod \"iptables-alerter-5wnp7\" (UID: \"80528350-3ca9-493f-94f7-39ddbf06ce11\") " pod="openshift-network-operator/iptables-alerter-5wnp7" Sep 17 18:02:57.649579 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.649556 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" Sep 17 18:02:57.657461 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.657439 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" Sep 17 18:02:57.670571 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.670554 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f87tt" Sep 17 18:02:57.676624 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.676607 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q7k8g" Sep 17 18:02:57.687116 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.687098 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h9q2n" Sep 17 18:02:57.693364 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.693342 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l529v" Sep 17 18:02:57.700809 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.700776 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5wnp7" Sep 17 18:02:57.708588 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.708569 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:02:57.716236 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:57.716222 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8hr8c" Sep 17 18:02:58.159489 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:58.159459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:02:58.159489 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:58.159492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:02:58.159710 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:58.159518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lv8j\" (UniqueName: \"kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j\") pod \"network-check-target-b4sz9\" (UID: \"e4f805c2-e759-4c84-88c4-5d6e5e34de8f\") " pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:02:58.159710 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:58.159638 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:02:58.159710 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:58.159646 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:02:58.159710 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:58.159644 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:02:58.159842 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:58.159726 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:02:58.159842 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:58.159742 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7lv8j for pod openshift-network-diagnostics/network-check-target-b4sz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:02:58.159842 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:58.159704 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret podName:0704f62c-880a-451b-851f-731517d14596 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:59.159689956 +0000 UTC m=+4.169877069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret") pod "global-pull-secret-syncer-l2fvr" (UID: "0704f62c-880a-451b-851f-731517d14596") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:02:58.159842 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:58.159799 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs podName:18d9adda-d73b-4456-83d8-360539054c24 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:59.159780034 +0000 UTC m=+4.169967150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs") pod "network-metrics-daemon-mt8dc" (UID: "18d9adda-d73b-4456-83d8-360539054c24") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:02:58.159842 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:58.159824 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j podName:e4f805c2-e759-4c84-88c4-5d6e5e34de8f nodeName:}" failed. No retries permitted until 2026-09-17 18:02:59.159812618 +0000 UTC m=+4.169999734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lv8j" (UniqueName: "kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j") pod "network-check-target-b4sz9" (UID: "e4f805c2-e759-4c84-88c4-5d6e5e34de8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:02:58.425977 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:58.425896 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:57:57 +0000 UTC" deadline="2028-03-07 16:58:00.610734227 +0000 UTC" Sep 17 18:02:58.425977 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:58.425924 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12886h55m2.184814052s" Sep 17 18:02:58.502046 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:58.502005 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2e6ff5_15fc_4af0_8095_8c7627bc7110.slice/crio-831dd5f06b0c20bf6cf1085e79ded9cf805b0cc2b0f3c3b0d33cb8b61f73f92e WatchSource:0}: Error finding container 831dd5f06b0c20bf6cf1085e79ded9cf805b0cc2b0f3c3b0d33cb8b61f73f92e: Status 404 returned error can't find the container with id 831dd5f06b0c20bf6cf1085e79ded9cf805b0cc2b0f3c3b0d33cb8b61f73f92e Sep 17 18:02:58.508135 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:58.508108 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e0a74f_1624_4aaa_b102_d15049b61cca.slice/crio-ad42bc84aaf87e1b7aef930468928a8bdc4fac27133d2a2ea1acdc2997e172ad WatchSource:0}: Error finding container ad42bc84aaf87e1b7aef930468928a8bdc4fac27133d2a2ea1acdc2997e172ad: Status 404 returned error can't find the container with id ad42bc84aaf87e1b7aef930468928a8bdc4fac27133d2a2ea1acdc2997e172ad Sep 17 18:02:58.509235 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:58.509214 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36401200_8357_4c9b_9f41_7a13eb41f824.slice/crio-db05d62733df10181368d37451171cbde5948aabc4a0d457abc5428eed9d3808 WatchSource:0}: Error finding container db05d62733df10181368d37451171cbde5948aabc4a0d457abc5428eed9d3808: Status 404 returned error can't find the container with id db05d62733df10181368d37451171cbde5948aabc4a0d457abc5428eed9d3808 Sep 17 18:02:58.510165 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:58.510123 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1b7fa0_4d80_496a_a4ce_02a720060731.slice/crio-1692ea4323f210218566ff835b960c33de60bd206a409fd7ea082a9737a36a58 WatchSource:0}: Error finding container 1692ea4323f210218566ff835b960c33de60bd206a409fd7ea082a9737a36a58: Status 404 returned error can't find the container with id 1692ea4323f210218566ff835b960c33de60bd206a409fd7ea082a9737a36a58 Sep 17 18:02:58.511923 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:58.511340 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80528350_3ca9_493f_94f7_39ddbf06ce11.slice/crio-35dd2763eec75e0bbcc754badc7361c867b075b62da7ba5fdf35bd36bc991324 WatchSource:0}: Error finding container 35dd2763eec75e0bbcc754badc7361c867b075b62da7ba5fdf35bd36bc991324: Status 404 returned error can't find the container with id 35dd2763eec75e0bbcc754badc7361c867b075b62da7ba5fdf35bd36bc991324 Sep 17 18:02:58.512281 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:58.512248 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27232ef_6a25_479f_b137_8ae12d97efd6.slice/crio-bedc9edb7ceb10ac766b7f50dc8faf6c54197d971f3ff3e40a6c87165c431af4 WatchSource:0}: Error finding container bedc9edb7ceb10ac766b7f50dc8faf6c54197d971f3ff3e40a6c87165c431af4: Status 404 returned error can't find the container with id bedc9edb7ceb10ac766b7f50dc8faf6c54197d971f3ff3e40a6c87165c431af4 Sep 17 18:02:58.513565 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:02:58.513138 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73423cd_ef53_471d_8b77_c72793a394b8.slice/crio-74de841991cf74be43f2f6cc6f74254f46eddb9887589e3fa8ca6473d93911ca WatchSource:0}: Error finding container 74de841991cf74be43f2f6cc6f74254f46eddb9887589e3fa8ca6473d93911ca: Status 404 returned error can't find the container with id 74de841991cf74be43f2f6cc6f74254f46eddb9887589e3fa8ca6473d93911ca Sep 17 18:02:59.167791 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.167684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:02:59.167791 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.167730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:02:59.167791 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.167765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lv8j\" (UniqueName: \"kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j\") pod \"network-check-target-b4sz9\" (UID: \"e4f805c2-e759-4c84-88c4-5d6e5e34de8f\") " pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:02:59.168050 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:59.167937 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:02:59.168050 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:59.167955 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:02:59.168050 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:59.167968 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7lv8j for pod openshift-network-diagnostics/network-check-target-b4sz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:02:59.168183 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:59.168095 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j podName:e4f805c2-e759-4c84-88c4-5d6e5e34de8f nodeName:}" failed. No retries permitted until 2026-09-17 18:03:01.168074729 +0000 UTC m=+6.178261854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lv8j" (UniqueName: "kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j") pod "network-check-target-b4sz9" (UID: "e4f805c2-e759-4c84-88c4-5d6e5e34de8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:02:59.168614 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:59.168503 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:02:59.168614 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:59.168577 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret podName:0704f62c-880a-451b-851f-731517d14596 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:01.168561348 +0000 UTC m=+6.178748486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret") pod "global-pull-secret-syncer-l2fvr" (UID: "0704f62c-880a-451b-851f-731517d14596") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:02:59.168792 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:59.168632 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:02:59.168792 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:59.168665 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs podName:18d9adda-d73b-4456-83d8-360539054c24 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:01.168654028 +0000 UTC m=+6.178841143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs") pod "network-metrics-daemon-mt8dc" (UID: "18d9adda-d73b-4456-83d8-360539054c24") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:02:59.426460 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.426356 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:57:57 +0000 UTC" deadline="2028-05-13 17:30:46.70336648 +0000 UTC" Sep 17 18:02:59.426460 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.426419 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14495h27m47.276952096s" Sep 17 18:02:59.447031 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.447004 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:02:59.447154 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:59.447128 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:02:59.447568 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.447548 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:02:59.447690 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:59.447669 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:02:59.447751 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.447739 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:02:59.447832 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:02:59.447813 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:02:59.466271 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.466203 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f87tt" event={"ID":"21e09d04-55d0-4833-8f9c-ec78ea8beffc","Type":"ContainerStarted","Data":"6eb88b39e2ef0544c8cdbacd89ead19c98900f84388cb53f7e75ef3c9ddfcf26"} Sep 17 18:02:59.473053 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.473022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" event={"ID":"d18622155b9da2257267bc123b9432af","Type":"ContainerStarted","Data":"aacdaa0329bab733aa358cb947ca0397a716521b2b76eb6d99f9c4440a39e43f"} Sep 17 18:02:59.485971 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.485944 2564 generic.go:356] "Generic (PLEG): container finished" podID="cf72d80b110564496492d5b5b6546081" containerID="adbf6c7128be42965d8b1e86f245290cf67ab72d4fecb10ef2cd0571e5936bde" exitCode=0 Sep 17 18:02:59.486075 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.486021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" event={"ID":"cf72d80b110564496492d5b5b6546081","Type":"ContainerDied","Data":"adbf6c7128be42965d8b1e86f245290cf67ab72d4fecb10ef2cd0571e5936bde"} Sep 17 18:02:59.495147 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.495097 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" podStartSLOduration=3.495080851 podStartE2EDuration="3.495080851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:59.494171074 +0000 UTC m=+4.504358210" watchObservedRunningTime="2026-09-17 18:02:59.495080851 +0000 UTC m=+4.505267987" Sep 17 18:02:59.503114 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.503066 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" event={"ID":"b27232ef-6a25-479f-b137-8ae12d97efd6","Type":"ContainerStarted","Data":"bedc9edb7ceb10ac766b7f50dc8faf6c54197d971f3ff3e40a6c87165c431af4"} Sep 17 18:02:59.514220 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.514196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5wnp7" event={"ID":"80528350-3ca9-493f-94f7-39ddbf06ce11","Type":"ContainerStarted","Data":"35dd2763eec75e0bbcc754badc7361c867b075b62da7ba5fdf35bd36bc991324"} Sep 17 18:02:59.523241 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.523215 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l529v" event={"ID":"6d1b7fa0-4d80-496a-a4ce-02a720060731","Type":"ContainerStarted","Data":"1692ea4323f210218566ff835b960c33de60bd206a409fd7ea082a9737a36a58"} Sep 17 18:02:59.532233 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.532206 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" event={"ID":"36401200-8357-4c9b-9f41-7a13eb41f824","Type":"ContainerStarted","Data":"db05d62733df10181368d37451171cbde5948aabc4a0d457abc5428eed9d3808"} Sep 17 18:02:59.539418 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.539392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7k8g" event={"ID":"40e0a74f-1624-4aaa-b102-d15049b61cca","Type":"ContainerStarted","Data":"ad42bc84aaf87e1b7aef930468928a8bdc4fac27133d2a2ea1acdc2997e172ad"} Sep 17 18:02:59.556696 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.556670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" event={"ID":"9c2e6ff5-15fc-4af0-8095-8c7627bc7110","Type":"ContainerStarted","Data":"831dd5f06b0c20bf6cf1085e79ded9cf805b0cc2b0f3c3b0d33cb8b61f73f92e"} Sep 17 18:02:59.583012 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.582942 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8hr8c" event={"ID":"91d98768-2d1f-4172-a9c7-f782aa9056ac","Type":"ContainerStarted","Data":"8e22eaeb073adc8951a8594dc88fdfb1a2335ffdf38f65b30c5f252b8d94096e"} Sep 17 18:02:59.601349 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:02:59.601319 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h9q2n" event={"ID":"b73423cd-ef53-471d-8b77-c72793a394b8","Type":"ContainerStarted","Data":"74de841991cf74be43f2f6cc6f74254f46eddb9887589e3fa8ca6473d93911ca"} Sep 17 18:03:00.633566 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:00.633483 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" event={"ID":"cf72d80b110564496492d5b5b6546081","Type":"ContainerStarted","Data":"7a94c56dcca6e178462bb6406b1eb7576124fbbfab2035c26b8f011d0ada2ee4"} Sep 17 18:03:01.184425 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:01.184380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lv8j\" (UniqueName: \"kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j\") pod \"network-check-target-b4sz9\" (UID: \"e4f805c2-e759-4c84-88c4-5d6e5e34de8f\") " pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:01.184629 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:01.184466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:01.184629 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:01.184517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:01.184741 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:01.184655 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:01.184741 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:01.184722 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs podName:18d9adda-d73b-4456-83d8-360539054c24 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:05.184700878 +0000 UTC m=+10.194887997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs") pod "network-metrics-daemon-mt8dc" (UID: "18d9adda-d73b-4456-83d8-360539054c24") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:01.185150 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:01.185130 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:03:01.185223 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:01.185177 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:03:01.185223 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:01.185191 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7lv8j for pod openshift-network-diagnostics/network-check-target-b4sz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:01.185331 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:01.185240 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j podName:e4f805c2-e759-4c84-88c4-5d6e5e34de8f nodeName:}" failed. No retries permitted until 2026-09-17 18:03:05.185226837 +0000 UTC m=+10.195413957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lv8j" (UniqueName: "kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j") pod "network-check-target-b4sz9" (UID: "e4f805c2-e759-4c84-88c4-5d6e5e34de8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:01.185331 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:01.185301 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:01.185438 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:01.185336 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret podName:0704f62c-880a-451b-851f-731517d14596 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:05.185325828 +0000 UTC m=+10.195512945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret") pod "global-pull-secret-syncer-l2fvr" (UID: "0704f62c-880a-451b-851f-731517d14596") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:01.445101 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:01.445024 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:01.445255 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:01.445165 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:01.445686 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:01.445546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:01.445686 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:01.445642 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:01.446072 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:01.445947 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:01.446072 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:01.446039 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:03.445769 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:03.445051 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:03.445769 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:03.445164 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:03.445769 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:03.445198 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:03.445769 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:03.445292 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:03.445769 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:03.445695 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:03.445769 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:03.445739 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:05.218013 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:05.217975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:05.218013 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:05.218027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:05.218558 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:05.218066 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lv8j\" (UniqueName: \"kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j\") pod \"network-check-target-b4sz9\" (UID: \"e4f805c2-e759-4c84-88c4-5d6e5e34de8f\") " pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:05.218558 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:05.218192 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:03:05.218558 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:05.218210 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:03:05.218558 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:05.218221 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7lv8j for pod openshift-network-diagnostics/network-check-target-b4sz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:05.218558 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:05.218274 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j podName:e4f805c2-e759-4c84-88c4-5d6e5e34de8f nodeName:}" failed. No retries permitted until 2026-09-17 18:03:13.218256332 +0000 UTC m=+18.228443448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lv8j" (UniqueName: "kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j") pod "network-check-target-b4sz9" (UID: "e4f805c2-e759-4c84-88c4-5d6e5e34de8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:05.218823 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:05.218650 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:05.218823 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:05.218700 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret podName:0704f62c-880a-451b-851f-731517d14596 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:13.218685744 +0000 UTC m=+18.228872872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret") pod "global-pull-secret-syncer-l2fvr" (UID: "0704f62c-880a-451b-851f-731517d14596") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:05.218823 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:05.218757 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:05.218823 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:05.218792 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs podName:18d9adda-d73b-4456-83d8-360539054c24 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:13.218779599 +0000 UTC m=+18.228966718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs") pod "network-metrics-daemon-mt8dc" (UID: "18d9adda-d73b-4456-83d8-360539054c24") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:05.447085 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:05.446218 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:05.447085 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:05.446411 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:05.447085 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:05.446600 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:05.447085 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:05.446654 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:05.447085 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:05.446667 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:05.447085 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:05.446800 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:07.445419 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:07.445386 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:07.445857 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:07.445496 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:07.445857 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:07.445827 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:07.445950 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:07.445916 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:07.446041 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:07.446029 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:07.446108 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:07.446092 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:09.445674 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:09.445440 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:09.446074 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:09.445452 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:09.446074 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:09.445506 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:09.446074 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:09.445785 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:09.446074 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:09.445882 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:09.446074 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:09.445978 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:11.444853 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:11.444819 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:11.445294 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:11.444946 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:11.445294 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:11.445064 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:11.445294 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:11.445172 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:11.445294 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:11.445241 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:11.445515 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:11.445297 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:13.276387 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:13.276338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:13.276387 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:13.276385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:13.277004 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:13.276427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lv8j\" (UniqueName: \"kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j\") pod \"network-check-target-b4sz9\" (UID: \"e4f805c2-e759-4c84-88c4-5d6e5e34de8f\") " pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:13.277004 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:13.276503 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:13.277004 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:13.276565 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:13.277004 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:13.276575 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:03:13.277004 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:13.276593 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret podName:0704f62c-880a-451b-851f-731517d14596 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:29.276572463 +0000 UTC m=+34.286759591 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret") pod "global-pull-secret-syncer-l2fvr" (UID: "0704f62c-880a-451b-851f-731517d14596") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:13.277004 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:13.276594 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:03:13.277004 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:13.276611 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7lv8j for pod openshift-network-diagnostics/network-check-target-b4sz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:13.277004 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:13.276611 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs podName:18d9adda-d73b-4456-83d8-360539054c24 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:29.276602686 +0000 UTC m=+34.286789806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs") pod "network-metrics-daemon-mt8dc" (UID: "18d9adda-d73b-4456-83d8-360539054c24") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:13.277004 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:13.276669 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j podName:e4f805c2-e759-4c84-88c4-5d6e5e34de8f nodeName:}" failed. No retries permitted until 2026-09-17 18:03:29.276652097 +0000 UTC m=+34.286839215 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lv8j" (UniqueName: "kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j") pod "network-check-target-b4sz9" (UID: "e4f805c2-e759-4c84-88c4-5d6e5e34de8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:13.444844 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:13.444818 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:13.444997 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:13.444820 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:13.444997 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:13.444940 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:13.445077 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:13.445042 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:13.445077 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:13.444820 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:13.445151 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:13.445130 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:15.446074 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:15.446043 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:15.446450 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:15.446145 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:15.446450 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:15.446204 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:15.446450 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:15.446268 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:15.446450 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:15.446355 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:15.446620 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:15.446456 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:16.663888 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.663670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l529v" event={"ID":"6d1b7fa0-4d80-496a-a4ce-02a720060731","Type":"ContainerStarted","Data":"8c02c2654689938b3bfc049689938b76c7a6cd5fdc08acb6da4b6a62d43e6fe2"} Sep 17 18:03:16.665264 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.665242 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" event={"ID":"36401200-8357-4c9b-9f41-7a13eb41f824","Type":"ContainerStarted","Data":"4026cf10bada6947bf50295cfda4c9d6e0df1108f9fc80bcc5e4b383499f3206"} Sep 17 18:03:16.666798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.666768 2564 generic.go:356] "Generic (PLEG): container finished" podID="40e0a74f-1624-4aaa-b102-d15049b61cca" containerID="3e27931bb56c0b7363fce5b6dedc5b3e3c035b8cfe3215b2cd7a9d08267bf926" exitCode=0 Sep 17 18:03:16.666917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.666844 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7k8g" event={"ID":"40e0a74f-1624-4aaa-b102-d15049b61cca","Type":"ContainerDied","Data":"3e27931bb56c0b7363fce5b6dedc5b3e3c035b8cfe3215b2cd7a9d08267bf926"} Sep 17 18:03:16.670974 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.670954 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" event={"ID":"9c2e6ff5-15fc-4af0-8095-8c7627bc7110","Type":"ContainerStarted","Data":"794fc3fe18293d903c0ded1f0bbfa33f44162329c2c3e5cc0dd2c5aa120361e3"} Sep 17 18:03:16.671050 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.671020 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" event={"ID":"9c2e6ff5-15fc-4af0-8095-8c7627bc7110","Type":"ContainerStarted","Data":"0aabad845e1b55449f751ea667862aac6cfc1065159185f027d14664052a6121"} Sep 17 18:03:16.671095 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.671063 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" event={"ID":"9c2e6ff5-15fc-4af0-8095-8c7627bc7110","Type":"ContainerStarted","Data":"cb72ff4681145fb5cef24075a1e42d95b9d7c44c1bf2c58e7194b874cdff20f1"} Sep 17 18:03:16.671131 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.671102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" event={"ID":"9c2e6ff5-15fc-4af0-8095-8c7627bc7110","Type":"ContainerStarted","Data":"27dedf711fb04fb902f87f23067c3bdfa60a1b09dc3885357044b2532e02c4b7"} Sep 17 18:03:16.671184 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.671130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" event={"ID":"9c2e6ff5-15fc-4af0-8095-8c7627bc7110","Type":"ContainerStarted","Data":"bffcf5f000a99540c78a4e6dbbc18085f08befa9e5008b90be33f0b0367d78c8"} Sep 17 18:03:16.671184 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.671164 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" event={"ID":"9c2e6ff5-15fc-4af0-8095-8c7627bc7110","Type":"ContainerStarted","Data":"670624e43b79c802d1d909e6034a3f6eb95390d18e41b68837c99b0381a89a7d"} Sep 17 18:03:16.672593 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.672570 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8hr8c" event={"ID":"91d98768-2d1f-4172-a9c7-f782aa9056ac","Type":"ContainerStarted","Data":"e2f7c5075f793a63ef47762ab8268c39e24ae61014db82e4cc277064b1e4ac4d"} Sep 17 18:03:16.674159 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.674131 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h9q2n" event={"ID":"b73423cd-ef53-471d-8b77-c72793a394b8","Type":"ContainerStarted","Data":"e7dbd187f7924494029ff304188b0aa3c9832d5ff07075f06ba0edfcf9282da0"} Sep 17 18:03:16.675853 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.675835 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f87tt" event={"ID":"21e09d04-55d0-4833-8f9c-ec78ea8beffc","Type":"ContainerStarted","Data":"8c8c3781e3e8d8defade84d1acb7264cec2fc94680423a3d513b4e387a243503"} Sep 17 18:03:16.677254 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.677234 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" event={"ID":"b27232ef-6a25-479f-b137-8ae12d97efd6","Type":"ContainerStarted","Data":"2d7a877f5fd4102607070ea750870fdbed799b524ce2e9a9c931ed71eca7bebd"} Sep 17 18:03:16.685938 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.685902 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l529v" podStartSLOduration=4.115965243 podStartE2EDuration="21.685891999s" totalImagesPullingTime="17.569926756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:16.685864992 +0000 UTC m=+21.696052132" watchObservedRunningTime="2026-09-17 18:03:16.685891999 +0000 UTC m=+21.696079134" Sep 17 18:03:16.686263 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.686238 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" podStartSLOduration=20.686232537 podStartE2EDuration="20.686232537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:00.669735476 +0000 UTC m=+5.679922610" watchObservedRunningTime="2026-09-17 18:03:16.686232537 +0000 UTC m=+21.696419672" Sep 17 18:03:16.704107 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.704074 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h9q2n" podStartSLOduration=8.01289899 podStartE2EDuration="21.704058589s" totalImagesPullingTime="13.691159599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:16.704049042 +0000 UTC m=+21.714236179" watchObservedRunningTime="2026-09-17 18:03:16.704058589 +0000 UTC m=+21.714245726" Sep 17 18:03:16.725243 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.725212 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8hr8c" podStartSLOduration=4.221371961 podStartE2EDuration="21.725199146s" totalImagesPullingTime="17.503827185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:16.725160999 +0000 UTC m=+21.735348135" watchObservedRunningTime="2026-09-17 18:03:16.725199146 +0000 UTC m=+21.735386281" Sep 17 18:03:16.737611 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.737595 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f87tt" Sep 17 18:03:16.738059 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.738037 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f87tt" Sep 17 18:03:16.768979 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.768946 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k9dvm" podStartSLOduration=4.259212489 podStartE2EDuration="21.768933156s" totalImagesPullingTime="17.509720667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:16.74492019 +0000 UTC m=+21.755107328" watchObservedRunningTime="2026-09-17 18:03:16.768933156 +0000 UTC m=+21.779120291" Sep 17 18:03:16.769318 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:16.769293 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f87tt" podStartSLOduration=8.067965701 podStartE2EDuration="21.769287014s" totalImagesPullingTime="13.701321313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:16.769052683 +0000 UTC m=+21.779239819" watchObservedRunningTime="2026-09-17 18:03:16.769287014 +0000 UTC m=+21.779474149" Sep 17 18:03:17.445237 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:17.445216 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:17.445237 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:17.445225 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:17.445423 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:17.445228 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:17.445423 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:17.445307 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:17.445423 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:17.445388 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:17.445559 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:17.445449 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:17.655121 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:17.654893 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 18:03:17.680649 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:17.680620 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5wnp7" event={"ID":"80528350-3ca9-493f-94f7-39ddbf06ce11","Type":"ContainerStarted","Data":"f3d65bdf7fd9c4a9f8c7c696d7068d0ac74777d86d6ae2643eb20cafafe4751b"} Sep 17 18:03:17.682251 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:17.682231 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" event={"ID":"36401200-8357-4c9b-9f41-7a13eb41f824","Type":"ContainerStarted","Data":"9bc5fb91cf76d069c467b002604518a9c1f52797f30a7e9379e7f897968fb566"} Sep 17 18:03:17.682862 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:17.682844 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f87tt" Sep 17 18:03:17.682998 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:17.682984 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f87tt" Sep 17 18:03:17.722218 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:17.722178 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5wnp7" podStartSLOduration=5.213633038 podStartE2EDuration="22.722166412s" totalImagesPullingTime="17.508533374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:17.701344583 +0000 UTC m=+22.711531731" watchObservedRunningTime="2026-09-17 18:03:17.722166412 +0000 UTC m=+22.732353548" Sep 17 18:03:18.420999 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:18.420887 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:03:17.655119989Z","UUID":"f27918bd-a12f-4099-a388-41decd19f84a","Handler":null,"Name":"","Endpoint":""} Sep 17 18:03:18.422707 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:18.422632 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 18:03:18.422707 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:18.422669 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 18:03:19.444506 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:19.444477 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:19.444506 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:19.444501 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:19.444506 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:19.444514 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:19.445119 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:19.444631 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:19.445119 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:19.444706 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:19.445119 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:19.444795 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:19.693190 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:19.693154 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" event={"ID":"36401200-8357-4c9b-9f41-7a13eb41f824","Type":"ContainerStarted","Data":"19c7abdd1e79ae31b51d32ab2d6a80f121eb4623bb74aaaf716266f1a882f14b"} Sep 17 18:03:19.698030 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:19.697937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" event={"ID":"9c2e6ff5-15fc-4af0-8095-8c7627bc7110","Type":"ContainerStarted","Data":"1ee8522f5a060777618a8542cbd3b2535c019ebc914ebe73c080f4e6c6020f2c"} Sep 17 18:03:19.718192 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:19.718146 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwnpk" podStartSLOduration=4.977803788 podStartE2EDuration="24.718128352s" totalImagesPullingTime="19.740324564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:19.717229966 +0000 UTC m=+24.727417104" watchObservedRunningTime="2026-09-17 18:03:19.718128352 +0000 UTC m=+24.728315489" Sep 17 18:03:21.444455 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:21.444426 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:21.444455 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:21.444447 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:21.444866 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:21.444447 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:21.444866 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:21.444514 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:21.444866 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:21.444613 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:21.444866 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:21.444669 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:21.706558 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:21.706475 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" event={"ID":"9c2e6ff5-15fc-4af0-8095-8c7627bc7110","Type":"ContainerStarted","Data":"0302e4ed09b51b530da678ac048702a7d941dc12aa68bd761b4a9a51454d8a7d"} Sep 17 18:03:21.706829 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:21.706802 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:03:21.706917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:21.706882 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:03:21.725737 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:21.725713 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:03:21.752854 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:21.752809 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" podStartSLOduration=9.170805856 podStartE2EDuration="26.752796102s" totalImagesPullingTime="17.581990246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:21.752381876 +0000 UTC m=+26.762569013" watchObservedRunningTime="2026-09-17 18:03:21.752796102 +0000 UTC m=+26.762983238" Sep 17 18:03:22.709775 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:22.709746 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:03:22.725723 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:22.725702 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:03:22.798182 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:22.798158 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b4sz9"] Sep 17 18:03:22.798289 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:22.798260 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:22.798360 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:22.798339 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:22.801585 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:22.801561 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l2fvr"] Sep 17 18:03:22.801692 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:22.801682 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:22.801873 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:22.801817 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:22.803026 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:22.803002 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mt8dc"] Sep 17 18:03:22.803111 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:22.803096 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:22.803201 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:22.803183 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:24.445474 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:24.445414 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:24.445908 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:24.445511 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:24.445908 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:24.445521 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:24.445908 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:24.445556 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:24.445908 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:24.445631 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:24.445908 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:24.445716 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:24.715933 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:24.715862 2564 generic.go:356] "Generic (PLEG): container finished" podID="40e0a74f-1624-4aaa-b102-d15049b61cca" containerID="0112623f1da149b96e8550681442bf78c518e4a8cfa8832fb570c6430a16096a" exitCode=0 Sep 17 18:03:24.716060 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:24.715950 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7k8g" event={"ID":"40e0a74f-1624-4aaa-b102-d15049b61cca","Type":"ContainerDied","Data":"0112623f1da149b96e8550681442bf78c518e4a8cfa8832fb570c6430a16096a"} Sep 17 18:03:26.444649 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:26.444451 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:26.444946 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:26.444484 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:26.444946 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:26.444725 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:26.444946 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:26.444483 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:26.444946 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:26.444809 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:26.444946 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:26.444883 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:26.722627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:26.722558 2564 generic.go:356] "Generic (PLEG): container finished" podID="40e0a74f-1624-4aaa-b102-d15049b61cca" containerID="99a5c4a5296c0bb4a3f1875363cb9325f791db32757b998623dc538846bd9ba7" exitCode=0 Sep 17 18:03:26.722627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:26.722622 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7k8g" event={"ID":"40e0a74f-1624-4aaa-b102-d15049b61cca","Type":"ContainerDied","Data":"99a5c4a5296c0bb4a3f1875363cb9325f791db32757b998623dc538846bd9ba7"} Sep 17 18:03:28.444869 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.444842 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:28.445318 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:28.444936 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4sz9" podUID="e4f805c2-e759-4c84-88c4-5d6e5e34de8f" Sep 17 18:03:28.445318 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.444847 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:28.445318 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:28.445012 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt8dc" podUID="18d9adda-d73b-4456-83d8-360539054c24" Sep 17 18:03:28.445318 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.444846 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:28.445318 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:28.445144 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2fvr" podUID="0704f62c-880a-451b-851f-731517d14596" Sep 17 18:03:28.729380 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.729311 2564 generic.go:356] "Generic (PLEG): container finished" podID="40e0a74f-1624-4aaa-b102-d15049b61cca" containerID="b8ab338b7d7e07ddb3897f66c8e7effdf89804a76340ea95b492b86192ac9170" exitCode=0 Sep 17 18:03:28.729380 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.729369 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7k8g" event={"ID":"40e0a74f-1624-4aaa-b102-d15049b61cca","Type":"ContainerDied","Data":"b8ab338b7d7e07ddb3897f66c8e7effdf89804a76340ea95b492b86192ac9170"} Sep 17 18:03:28.797949 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.797932 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeReady" Sep 17 18:03:28.798043 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.798018 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 18:03:28.851313 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.851292 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9zglp"] Sep 17 18:03:28.888847 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.888816 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lcwnb"] Sep 17 18:03:28.888949 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.888936 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:28.891970 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.891952 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 18:03:28.892056 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.892024 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7dfk9\"" Sep 17 18:03:28.892194 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.892181 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 18:03:28.906139 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.906124 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9zglp"] Sep 17 18:03:28.906208 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.906154 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lcwnb"] Sep 17 18:03:28.906208 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.906203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lcwnb" Sep 17 18:03:28.908993 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.908972 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 18:03:28.909083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.908997 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fht8w\"" Sep 17 18:03:28.909083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.908982 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 18:03:28.909083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.908984 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 18:03:28.994462 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.994413 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:28.994462 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.994440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b8385b17-3d85-46f2-abbc-f92dd8cc9357-config-volume\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:28.994462 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.994455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d-cert\") pod \"ingress-canary-lcwnb\" (UID: \"d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d\") " pod="openshift-ingress-canary/ingress-canary-lcwnb" Sep 17 18:03:28.994657 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.994520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vlb\" (UniqueName: \"kubernetes.io/projected/d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d-kube-api-access-k7vlb\") pod \"ingress-canary-lcwnb\" (UID: \"d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d\") " pod="openshift-ingress-canary/ingress-canary-lcwnb" Sep 17 18:03:28.994657 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.994593 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8385b17-3d85-46f2-abbc-f92dd8cc9357-tmp-dir\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:28.994657 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:28.994611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgnrd\" (UniqueName: \"kubernetes.io/projected/b8385b17-3d85-46f2-abbc-f92dd8cc9357-kube-api-access-fgnrd\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:29.095664 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.095636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vlb\" (UniqueName: \"kubernetes.io/projected/d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d-kube-api-access-k7vlb\") pod \"ingress-canary-lcwnb\" (UID: \"d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d\") " pod="openshift-ingress-canary/ingress-canary-lcwnb" Sep 17 18:03:29.095764 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.095706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8385b17-3d85-46f2-abbc-f92dd8cc9357-tmp-dir\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:29.095764 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.095742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgnrd\" (UniqueName: \"kubernetes.io/projected/b8385b17-3d85-46f2-abbc-f92dd8cc9357-kube-api-access-fgnrd\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:29.095873 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.095779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:29.095873 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.095810 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b8385b17-3d85-46f2-abbc-f92dd8cc9357-config-volume\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:29.095873 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.095831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d-cert\") pod \"ingress-canary-lcwnb\" (UID: \"d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d\") " pod="openshift-ingress-canary/ingress-canary-lcwnb" Sep 17 18:03:29.096029 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:29.095892 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:03:29.096029 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:29.095957 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls podName:b8385b17-3d85-46f2-abbc-f92dd8cc9357 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:29.595943055 +0000 UTC m=+34.606130173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls") pod "dns-default-9zglp" (UID: "b8385b17-3d85-46f2-abbc-f92dd8cc9357") : secret "dns-default-metrics-tls" not found Sep 17 18:03:29.096029 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.095980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8385b17-3d85-46f2-abbc-f92dd8cc9357-tmp-dir\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:29.096310 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.096290 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b8385b17-3d85-46f2-abbc-f92dd8cc9357-config-volume\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:29.100327 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.100307 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d-cert\") pod \"ingress-canary-lcwnb\" (UID: \"d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d\") " pod="openshift-ingress-canary/ingress-canary-lcwnb" Sep 17 18:03:29.108036 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.108016 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgnrd\" (UniqueName: \"kubernetes.io/projected/b8385b17-3d85-46f2-abbc-f92dd8cc9357-kube-api-access-fgnrd\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:29.108618 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.108600 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vlb\" (UniqueName: \"kubernetes.io/projected/d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d-kube-api-access-k7vlb\") pod \"ingress-canary-lcwnb\" (UID: \"d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d\") " pod="openshift-ingress-canary/ingress-canary-lcwnb" Sep 17 18:03:29.213687 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.213668 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lcwnb" Sep 17 18:03:29.298127 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.297855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:29.298127 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.298108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:29.298324 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:29.298015 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:29.298324 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.298159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lv8j\" (UniqueName: \"kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j\") pod \"network-check-target-b4sz9\" (UID: \"e4f805c2-e759-4c84-88c4-5d6e5e34de8f\") " pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:29.298324 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:29.298241 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret podName:0704f62c-880a-451b-851f-731517d14596 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:01.298219227 +0000 UTC m=+66.308406346 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret") pod "global-pull-secret-syncer-l2fvr" (UID: "0704f62c-880a-451b-851f-731517d14596") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:29.298324 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:29.298244 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:29.298324 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:29.298263 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:03:29.298324 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:29.298281 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs podName:18d9adda-d73b-4456-83d8-360539054c24 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:01.298269781 +0000 UTC m=+66.308456923 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs") pod "network-metrics-daemon-mt8dc" (UID: "18d9adda-d73b-4456-83d8-360539054c24") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:29.298324 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:29.298282 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:03:29.298324 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:29.298294 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7lv8j for pod openshift-network-diagnostics/network-check-target-b4sz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:29.298324 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:29.298331 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j podName:e4f805c2-e759-4c84-88c4-5d6e5e34de8f nodeName:}" failed. No retries permitted until 2026-09-17 18:04:01.298319445 +0000 UTC m=+66.308506559 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lv8j" (UniqueName: "kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j") pod "network-check-target-b4sz9" (UID: "e4f805c2-e759-4c84-88c4-5d6e5e34de8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:29.544561 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.544507 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lcwnb"] Sep 17 18:03:29.550811 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:03:29.550779 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e5b8f7_05b4_4cf0_bb6e_8230dcfbad8d.slice/crio-78261cba8a54d1e97228aef4e8f500fb4a76143757fc454b841960351f95bbd8 WatchSource:0}: Error finding container 78261cba8a54d1e97228aef4e8f500fb4a76143757fc454b841960351f95bbd8: Status 404 returned error can't find the container with id 78261cba8a54d1e97228aef4e8f500fb4a76143757fc454b841960351f95bbd8 Sep 17 18:03:29.600390 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.600363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:29.600520 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:29.600503 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:03:29.600603 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:29.600584 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls podName:b8385b17-3d85-46f2-abbc-f92dd8cc9357 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:30.600563993 +0000 UTC m=+35.610751126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls") pod "dns-default-9zglp" (UID: "b8385b17-3d85-46f2-abbc-f92dd8cc9357") : secret "dns-default-metrics-tls" not found Sep 17 18:03:29.733543 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:29.733499 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lcwnb" event={"ID":"d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d","Type":"ContainerStarted","Data":"78261cba8a54d1e97228aef4e8f500fb4a76143757fc454b841960351f95bbd8"} Sep 17 18:03:30.444689 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:30.444497 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:03:30.444689 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:30.444552 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:03:30.444689 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:30.444557 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:03:30.447462 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:30.447437 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 18:03:30.448436 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:30.448200 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 18:03:30.448436 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:30.448231 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qc24z\"" Sep 17 18:03:30.448436 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:30.448232 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2v7h2\"" Sep 17 18:03:30.448436 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:30.448284 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 18:03:30.448436 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:30.448386 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 18:03:30.609048 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:30.609019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:30.609568 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:30.609172 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:03:30.609568 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:30.609243 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls podName:b8385b17-3d85-46f2-abbc-f92dd8cc9357 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:32.609220461 +0000 UTC m=+37.619407588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls") pod "dns-default-9zglp" (UID: "b8385b17-3d85-46f2-abbc-f92dd8cc9357") : secret "dns-default-metrics-tls" not found Sep 17 18:03:32.625467 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:32.625394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:32.626075 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:32.625514 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:03:32.626075 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:32.625591 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls podName:b8385b17-3d85-46f2-abbc-f92dd8cc9357 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:36.625577937 +0000 UTC m=+41.635765051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls") pod "dns-default-9zglp" (UID: "b8385b17-3d85-46f2-abbc-f92dd8cc9357") : secret "dns-default-metrics-tls" not found Sep 17 18:03:34.749146 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:34.749082 2564 generic.go:356] "Generic (PLEG): container finished" podID="40e0a74f-1624-4aaa-b102-d15049b61cca" containerID="38aa39ee3b4baf1e7a0c15c18f65cd594db9126e0f4f17f780aae07b14191fe5" exitCode=0 Sep 17 18:03:34.749701 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:34.749167 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7k8g" event={"ID":"40e0a74f-1624-4aaa-b102-d15049b61cca","Type":"ContainerDied","Data":"38aa39ee3b4baf1e7a0c15c18f65cd594db9126e0f4f17f780aae07b14191fe5"} Sep 17 18:03:34.750959 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:34.750928 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lcwnb" event={"ID":"d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d","Type":"ContainerStarted","Data":"1728db71d3d2c38b2d97b1026f56eaa1a760d1fd40df792ae8f3bc0fd53e07a5"} Sep 17 18:03:34.805426 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:34.805379 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lcwnb" podStartSLOduration=2.361179222 podStartE2EDuration="6.805342189s" totalImagesPullingTime="4.444162967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:34.804931723 +0000 UTC m=+39.815118859" watchObservedRunningTime="2026-09-17 18:03:34.805342189 +0000 UTC m=+39.815529326" Sep 17 18:03:35.760774 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:35.760741 2564 generic.go:356] "Generic (PLEG): container finished" podID="40e0a74f-1624-4aaa-b102-d15049b61cca" containerID="3f28d354f827e9fc4244c6daf27586b992f74c3250943b8959961125e77a6462" exitCode=0 Sep 17 18:03:35.761212 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:35.760823 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7k8g" event={"ID":"40e0a74f-1624-4aaa-b102-d15049b61cca","Type":"ContainerDied","Data":"3f28d354f827e9fc4244c6daf27586b992f74c3250943b8959961125e77a6462"} Sep 17 18:03:36.650388 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:36.650350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:36.650545 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:36.650505 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:03:36.650608 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:36.650592 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls podName:b8385b17-3d85-46f2-abbc-f92dd8cc9357 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:44.650572867 +0000 UTC m=+49.660759985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls") pod "dns-default-9zglp" (UID: "b8385b17-3d85-46f2-abbc-f92dd8cc9357") : secret "dns-default-metrics-tls" not found Sep 17 18:03:36.768236 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:36.768208 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7k8g" event={"ID":"40e0a74f-1624-4aaa-b102-d15049b61cca","Type":"ContainerStarted","Data":"4f58a34533877ee8442f98f577c0994cc7086bfb631583e9be306e8717e3c8d5"} Sep 17 18:03:36.800809 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:36.800755 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q7k8g" podStartSLOduration=9.443838032 podStartE2EDuration="41.800735243s" totalImagesPullingTime="32.356897211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:36.79918326 +0000 UTC m=+41.809370453" watchObservedRunningTime="2026-09-17 18:03:36.800735243 +0000 UTC m=+41.810922454" Sep 17 18:03:44.702299 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:44.702249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:03:44.702811 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:44.702423 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:03:44.702811 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:44.702503 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls podName:b8385b17-3d85-46f2-abbc-f92dd8cc9357 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:00.702484114 +0000 UTC m=+65.712671229 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls") pod "dns-default-9zglp" (UID: "b8385b17-3d85-46f2-abbc-f92dd8cc9357") : secret "dns-default-metrics-tls" not found Sep 17 18:03:54.728805 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:54.728769 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qx27j" Sep 17 18:03:56.778595 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.778561 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-9xz92"] Sep 17 18:03:56.781202 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.781182 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.782830 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.782804 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64665745fb-n4rxl"] Sep 17 18:03:56.784996 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.784975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.787785 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.787755 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 18:03:56.788079 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.788056 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 18:03:56.788325 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.788294 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 17 18:03:56.788728 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.788308 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 17 18:03:56.788873 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.788833 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vbcxb\"" Sep 17 18:03:56.791903 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.791442 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 18:03:56.791903 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.791794 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 17 18:03:56.791903 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.791862 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xfsgr\"" Sep 17 18:03:56.791903 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.791868 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 17 18:03:56.792160 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.792117 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 17 18:03:56.792215 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.792194 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 17 18:03:56.792582 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.792559 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 18:03:56.795894 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.795874 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 17 18:03:56.799128 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.799099 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-9xz92"] Sep 17 18:03:56.813348 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.813326 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64665745fb-n4rxl"] Sep 17 18:03:56.878719 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.878692 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3b9adda-9178-46dd-885b-9ffa426d7e03-tmp\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.878719 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.878720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-default-certificate\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.878846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.878742 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b9adda-9178-46dd-885b-9ffa426d7e03-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.878846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.878762 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c3b9adda-9178-46dd-885b-9ffa426d7e03-snapshots\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.878846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.878800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-stats-auth\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.878846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.878817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.878971 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.878866 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgtz6\" (UniqueName: \"kubernetes.io/projected/c3b9adda-9178-46dd-885b-9ffa426d7e03-kube-api-access-pgtz6\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.878971 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.878884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.878971 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.878904 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b9adda-9178-46dd-885b-9ffa426d7e03-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.878971 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.878919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b9adda-9178-46dd-885b-9ffa426d7e03-serving-cert\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.878971 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.878935 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxjmk\" (UniqueName: \"kubernetes.io/projected/8cc392b6-399b-4293-9c92-afc8dbf7d952-kube-api-access-xxjmk\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.979886 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.979860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgtz6\" (UniqueName: \"kubernetes.io/projected/c3b9adda-9178-46dd-885b-9ffa426d7e03-kube-api-access-pgtz6\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.979976 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.979889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.979976 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.979911 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b9adda-9178-46dd-885b-9ffa426d7e03-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.979976 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.979927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b9adda-9178-46dd-885b-9ffa426d7e03-serving-cert\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.979976 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.979945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxjmk\" (UniqueName: \"kubernetes.io/projected/8cc392b6-399b-4293-9c92-afc8dbf7d952-kube-api-access-xxjmk\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.980176 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:56.980056 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle podName:8cc392b6-399b-4293-9c92-afc8dbf7d952 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:57.480032401 +0000 UTC m=+62.490219518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle") pod "router-default-64665745fb-n4rxl" (UID: "8cc392b6-399b-4293-9c92-afc8dbf7d952") : configmap references non-existent config key: service-ca.crt Sep 17 18:03:56.980176 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.980087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3b9adda-9178-46dd-885b-9ffa426d7e03-tmp\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.980176 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.980123 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-default-certificate\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.980331 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.980254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b9adda-9178-46dd-885b-9ffa426d7e03-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.980331 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.980299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c3b9adda-9178-46dd-885b-9ffa426d7e03-snapshots\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.980331 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.980329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-stats-auth\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.980478 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.980377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.980574 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.980493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3b9adda-9178-46dd-885b-9ffa426d7e03-tmp\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.980574 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:56.980565 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:03:56.980678 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.980592 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b9adda-9178-46dd-885b-9ffa426d7e03-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.980731 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:56.980702 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs podName:8cc392b6-399b-4293-9c92-afc8dbf7d952 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:57.480680861 +0000 UTC m=+62.490868002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs") pod "router-default-64665745fb-n4rxl" (UID: "8cc392b6-399b-4293-9c92-afc8dbf7d952") : secret "router-metrics-certs-default" not found Sep 17 18:03:56.981149 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.981118 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c3b9adda-9178-46dd-885b-9ffa426d7e03-snapshots\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.981377 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.981357 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b9adda-9178-46dd-885b-9ffa426d7e03-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.983200 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.983174 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b9adda-9178-46dd-885b-9ffa426d7e03-serving-cert\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:56.983294 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.983205 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-default-certificate\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.983294 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.983247 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-stats-auth\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.990237 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.990210 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxjmk\" (UniqueName: \"kubernetes.io/projected/8cc392b6-399b-4293-9c92-afc8dbf7d952-kube-api-access-xxjmk\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:56.990450 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:56.990429 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgtz6\" (UniqueName: \"kubernetes.io/projected/c3b9adda-9178-46dd-885b-9ffa426d7e03-kube-api-access-pgtz6\") pod \"insights-operator-6f9c644dd9-9xz92\" (UID: \"c3b9adda-9178-46dd-885b-9ffa426d7e03\") " pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:57.094159 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:57.094137 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" Sep 17 18:03:57.377069 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:57.377002 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-9xz92"] Sep 17 18:03:57.380169 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:03:57.380146 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b9adda_9178_46dd_885b_9ffa426d7e03.slice/crio-fc9890a423699ed1d2ce05d4777e95d88dfaad715170c3e739922f8420b8794b WatchSource:0}: Error finding container fc9890a423699ed1d2ce05d4777e95d88dfaad715170c3e739922f8420b8794b: Status 404 returned error can't find the container with id fc9890a423699ed1d2ce05d4777e95d88dfaad715170c3e739922f8420b8794b Sep 17 18:03:57.485161 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:57.485132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:57.485259 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:57.485215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:57.485331 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:57.485318 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle podName:8cc392b6-399b-4293-9c92-afc8dbf7d952 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:58.485296212 +0000 UTC m=+63.495483340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle") pod "router-default-64665745fb-n4rxl" (UID: "8cc392b6-399b-4293-9c92-afc8dbf7d952") : configmap references non-existent config key: service-ca.crt Sep 17 18:03:57.485331 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:57.485318 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:03:57.485420 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:57.485358 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs podName:8cc392b6-399b-4293-9c92-afc8dbf7d952 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:58.485349565 +0000 UTC m=+63.495536679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs") pod "router-default-64665745fb-n4rxl" (UID: "8cc392b6-399b-4293-9c92-afc8dbf7d952") : secret "router-metrics-certs-default" not found Sep 17 18:03:57.818655 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:57.818625 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" event={"ID":"c3b9adda-9178-46dd-885b-9ffa426d7e03","Type":"ContainerStarted","Data":"fc9890a423699ed1d2ce05d4777e95d88dfaad715170c3e739922f8420b8794b"} Sep 17 18:03:58.492896 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:58.492862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:58.493062 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:58.492921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:03:58.493062 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:58.493016 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:03:58.493062 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:58.493060 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle podName:8cc392b6-399b-4293-9c92-afc8dbf7d952 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:00.493045777 +0000 UTC m=+65.503232909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle") pod "router-default-64665745fb-n4rxl" (UID: "8cc392b6-399b-4293-9c92-afc8dbf7d952") : configmap references non-existent config key: service-ca.crt Sep 17 18:03:58.493197 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:03:58.493083 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs podName:8cc392b6-399b-4293-9c92-afc8dbf7d952 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:00.493073138 +0000 UTC m=+65.503260253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs") pod "router-default-64665745fb-n4rxl" (UID: "8cc392b6-399b-4293-9c92-afc8dbf7d952") : secret "router-metrics-certs-default" not found Sep 17 18:03:59.825541 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:59.825492 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" event={"ID":"c3b9adda-9178-46dd-885b-9ffa426d7e03","Type":"ContainerStarted","Data":"99531cf29e5758f6e2a7eb70df157ba0ee5dd370692a99bbe4e4bb2846387d69"} Sep 17 18:03:59.849802 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:03:59.849758 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" podStartSLOduration=1.885531445 podStartE2EDuration="3.849742346s" totalImagesPullingTime="1.964210901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:59.846799202 +0000 UTC m=+64.856986338" watchObservedRunningTime="2026-09-17 18:03:59.849742346 +0000 UTC m=+64.859929472" Sep 17 18:04:00.508253 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.508219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:00.508399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.508306 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:00.508441 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:00.508390 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle podName:8cc392b6-399b-4293-9c92-afc8dbf7d952 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:04.508367018 +0000 UTC m=+69.518554152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle") pod "router-default-64665745fb-n4rxl" (UID: "8cc392b6-399b-4293-9c92-afc8dbf7d952") : configmap references non-existent config key: service-ca.crt Sep 17 18:04:00.508441 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:00.508405 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:04:00.508511 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:00.508444 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs podName:8cc392b6-399b-4293-9c92-afc8dbf7d952 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:04.508432076 +0000 UTC m=+69.518619189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs") pod "router-default-64665745fb-n4rxl" (UID: "8cc392b6-399b-4293-9c92-afc8dbf7d952") : secret "router-metrics-certs-default" not found Sep 17 18:04:00.629803 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.629772 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-22vs2"] Sep 17 18:04:00.631921 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.631905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:00.636211 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.636192 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-87jlz\"" Sep 17 18:04:00.637096 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.637066 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 18:04:00.637217 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.637199 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 18:04:00.682667 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.682645 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-22vs2"] Sep 17 18:04:00.709362 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.709339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:04:00.709460 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:00.709449 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:04:00.709501 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:00.709492 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls podName:b8385b17-3d85-46f2-abbc-f92dd8cc9357 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:32.709480964 +0000 UTC m=+97.719668078 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls") pod "dns-default-9zglp" (UID: "b8385b17-3d85-46f2-abbc-f92dd8cc9357") : secret "dns-default-metrics-tls" not found Sep 17 18:04:00.810252 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.810232 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99fc2ce8-e335-4833-81b3-9c99d1897474-crio-socket\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:00.810333 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.810285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpvns\" (UniqueName: \"kubernetes.io/projected/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-api-access-zpvns\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:00.810333 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.810314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99fc2ce8-e335-4833-81b3-9c99d1897474-data-volume\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:00.810418 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.810342 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:00.810418 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.810362 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:00.911346 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.911320 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:00.911677 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.911356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:00.911677 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.911400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99fc2ce8-e335-4833-81b3-9c99d1897474-crio-socket\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:00.911677 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:00.911438 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:04:00.911677 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.911467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvns\" (UniqueName: \"kubernetes.io/projected/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-api-access-zpvns\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:00.911677 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:00.911490 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm podName:99fc2ce8-e335-4833-81b3-9c99d1897474 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:01.411472448 +0000 UTC m=+66.421659561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22vs2" (UID: "99fc2ce8-e335-4833-81b3-9c99d1897474") : configmap "kube-rbac-proxy" not found Sep 17 18:04:00.911677 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:00.911546 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:04:00.911677 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.911550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99fc2ce8-e335-4833-81b3-9c99d1897474-data-volume\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:00.911677 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.911547 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99fc2ce8-e335-4833-81b3-9c99d1897474-crio-socket\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:00.911677 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:00.911596 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls podName:99fc2ce8-e335-4833-81b3-9c99d1897474 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:01.411581661 +0000 UTC m=+66.421768792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22vs2" (UID: "99fc2ce8-e335-4833-81b3-9c99d1897474") : secret "insights-runtime-extractor-tls" not found Sep 17 18:04:00.911990 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.911834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99fc2ce8-e335-4833-81b3-9c99d1897474-data-volume\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:00.929985 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:00.929962 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvns\" (UniqueName: \"kubernetes.io/projected/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-api-access-zpvns\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:01.314131 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.314106 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:04:01.314242 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.314138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:04:01.314242 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.314177 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lv8j\" (UniqueName: \"kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j\") pod \"network-check-target-b4sz9\" (UID: \"e4f805c2-e759-4c84-88c4-5d6e5e34de8f\") " pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:04:01.317501 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.317483 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 18:04:01.317607 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.317564 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 18:04:01.317692 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.317673 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 18:04:01.325024 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:01.324995 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 18:04:01.325081 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:01.325065 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs podName:18d9adda-d73b-4456-83d8-360539054c24 nodeName:}" failed. No retries permitted until 2026-09-17 18:05:05.325049031 +0000 UTC m=+130.335236147 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs") pod "network-metrics-daemon-mt8dc" (UID: "18d9adda-d73b-4456-83d8-360539054c24") : secret "metrics-daemon-secret" not found Sep 17 18:04:01.327345 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.327317 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 18:04:01.327430 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.327347 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0704f62c-880a-451b-851f-731517d14596-original-pull-secret\") pod \"global-pull-secret-syncer-l2fvr\" (UID: \"0704f62c-880a-451b-851f-731517d14596\") " pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:04:01.338298 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.338270 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lv8j\" (UniqueName: \"kubernetes.io/projected/e4f805c2-e759-4c84-88c4-5d6e5e34de8f-kube-api-access-7lv8j\") pod \"network-check-target-b4sz9\" (UID: \"e4f805c2-e759-4c84-88c4-5d6e5e34de8f\") " pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:04:01.361273 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.361253 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2v7h2\"" Sep 17 18:04:01.365873 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.365859 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2fvr" Sep 17 18:04:01.368868 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.368854 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:04:01.415310 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.415266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:01.415481 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.415417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:01.415620 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:01.415518 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:04:01.415700 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:01.415653 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm podName:99fc2ce8-e335-4833-81b3-9c99d1897474 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:02.415632961 +0000 UTC m=+67.425820095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22vs2" (UID: "99fc2ce8-e335-4833-81b3-9c99d1897474") : configmap "kube-rbac-proxy" not found Sep 17 18:04:01.415814 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:01.415790 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:04:01.415870 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:01.415861 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls podName:99fc2ce8-e335-4833-81b3-9c99d1897474 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:02.41584524 +0000 UTC m=+67.426032373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22vs2" (UID: "99fc2ce8-e335-4833-81b3-9c99d1897474") : secret "insights-runtime-extractor-tls" not found Sep 17 18:04:01.654809 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.654717 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l2fvr"] Sep 17 18:04:01.655413 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.655389 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b4sz9"] Sep 17 18:04:01.658438 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:01.658405 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0704f62c_880a_451b_851f_731517d14596.slice/crio-3eb2448974726508242261ac337617d72b845704670210b22aa79e0b59db7994 WatchSource:0}: Error finding container 3eb2448974726508242261ac337617d72b845704670210b22aa79e0b59db7994: Status 404 returned error can't find the container with id 3eb2448974726508242261ac337617d72b845704670210b22aa79e0b59db7994 Sep 17 18:04:01.659050 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:01.659030 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f805c2_e759_4c84_88c4_5d6e5e34de8f.slice/crio-bb40c55ebe8cdd7f7253829f99d28101328d6ffc60a7f00ab6e198ce15168655 WatchSource:0}: Error finding container bb40c55ebe8cdd7f7253829f99d28101328d6ffc60a7f00ab6e198ce15168655: Status 404 returned error can't find the container with id bb40c55ebe8cdd7f7253829f99d28101328d6ffc60a7f00ab6e198ce15168655 Sep 17 18:04:01.831676 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.831645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b4sz9" event={"ID":"e4f805c2-e759-4c84-88c4-5d6e5e34de8f","Type":"ContainerStarted","Data":"bb40c55ebe8cdd7f7253829f99d28101328d6ffc60a7f00ab6e198ce15168655"} Sep 17 18:04:01.832795 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:01.832773 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l2fvr" event={"ID":"0704f62c-880a-451b-851f-731517d14596","Type":"ContainerStarted","Data":"3eb2448974726508242261ac337617d72b845704670210b22aa79e0b59db7994"} Sep 17 18:04:02.236428 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:02.236396 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h9q2n_b73423cd-ef53-471d-8b77-c72793a394b8/dns-node-resolver/0.log" Sep 17 18:04:02.424313 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:02.424115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:02.424313 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:02.424165 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:02.424313 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:02.424272 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:04:02.424596 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:02.424356 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:04:02.424596 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:02.424429 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm podName:99fc2ce8-e335-4833-81b3-9c99d1897474 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:04.424408598 +0000 UTC m=+69.434595735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22vs2" (UID: "99fc2ce8-e335-4833-81b3-9c99d1897474") : configmap "kube-rbac-proxy" not found Sep 17 18:04:02.424861 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:02.424833 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls podName:99fc2ce8-e335-4833-81b3-9c99d1897474 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:04.424810192 +0000 UTC m=+69.434997331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22vs2" (UID: "99fc2ce8-e335-4833-81b3-9c99d1897474") : secret "insights-runtime-extractor-tls" not found Sep 17 18:04:03.032701 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:03.032671 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8hr8c_91d98768-2d1f-4172-a9c7-f782aa9056ac/node-ca/0.log" Sep 17 18:04:04.031918 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:04.031841 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lcwnb_d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d/serve-healthcheck-canary/0.log" Sep 17 18:04:04.439458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:04.439418 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:04.439651 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:04.439561 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:04:04.439651 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:04.439573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:04.439651 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:04.439636 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls podName:99fc2ce8-e335-4833-81b3-9c99d1897474 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:08.439615016 +0000 UTC m=+73.449802133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22vs2" (UID: "99fc2ce8-e335-4833-81b3-9c99d1897474") : secret "insights-runtime-extractor-tls" not found Sep 17 18:04:04.439818 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:04.439663 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:04:04.439818 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:04.439739 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm podName:99fc2ce8-e335-4833-81b3-9c99d1897474 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:08.439722161 +0000 UTC m=+73.449909276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22vs2" (UID: "99fc2ce8-e335-4833-81b3-9c99d1897474") : configmap "kube-rbac-proxy" not found Sep 17 18:04:04.540846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:04.540815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:04.541012 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:04.540897 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:04.541012 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:04.540989 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle podName:8cc392b6-399b-4293-9c92-afc8dbf7d952 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:12.540968669 +0000 UTC m=+77.551155802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle") pod "router-default-64665745fb-n4rxl" (UID: "8cc392b6-399b-4293-9c92-afc8dbf7d952") : configmap references non-existent config key: service-ca.crt Sep 17 18:04:04.541120 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:04.541020 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:04:04.541120 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:04.541072 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs podName:8cc392b6-399b-4293-9c92-afc8dbf7d952 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:12.541056723 +0000 UTC m=+77.551243837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs") pod "router-default-64665745fb-n4rxl" (UID: "8cc392b6-399b-4293-9c92-afc8dbf7d952") : secret "router-metrics-certs-default" not found Sep 17 18:04:06.847561 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:06.847506 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l2fvr" event={"ID":"0704f62c-880a-451b-851f-731517d14596","Type":"ContainerStarted","Data":"4f11dd7c517a72cbfad8cc866069502d5190665b819f30af88f7aa89bb396eeb"} Sep 17 18:04:06.849082 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:06.849057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b4sz9" event={"ID":"e4f805c2-e759-4c84-88c4-5d6e5e34de8f","Type":"ContainerStarted","Data":"844c5137a2f8faabb514fcc7d7a55244caee3c6639aef45b407ca65c18c1bb79"} Sep 17 18:04:06.849196 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:06.849119 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:04:06.869222 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:06.869161 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l2fvr" podStartSLOduration=67.123690306 podStartE2EDuration="1m11.869150242s" totalImagesPullingTime="4.745459936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:06.868752942 +0000 UTC m=+71.878940077" watchObservedRunningTime="2026-09-17 18:04:06.869150242 +0000 UTC m=+71.879337378" Sep 17 18:04:08.450201 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:08.450144 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b4sz9" podStartSLOduration=68.709138298 podStartE2EDuration="1m13.450098417s" totalImagesPullingTime="4.740960119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:06.895447815 +0000 UTC m=+71.905634951" watchObservedRunningTime="2026-09-17 18:04:08.450098417 +0000 UTC m=+73.460285613" Sep 17 18:04:08.451141 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:08.451115 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-pgpj7"] Sep 17 18:04:08.453372 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:08.453353 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-pgpj7" Sep 17 18:04:08.456268 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:08.456247 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-s8gxz\"" Sep 17 18:04:08.469756 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:08.469732 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-pgpj7"] Sep 17 18:04:08.472105 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:08.472079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:08.472202 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:08.472118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6996l\" (UniqueName: \"kubernetes.io/projected/c30c2ea2-d35b-4ac0-b0c7-d4e059951efc-kube-api-access-6996l\") pod \"network-check-source-7cbc45fddc-pgpj7\" (UID: \"c30c2ea2-d35b-4ac0-b0c7-d4e059951efc\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-pgpj7" Sep 17 18:04:08.472202 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:08.472157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:08.472202 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:08.472195 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:04:08.472310 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:08.472273 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm podName:99fc2ce8-e335-4833-81b3-9c99d1897474 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:16.472255371 +0000 UTC m=+81.482442490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22vs2" (UID: "99fc2ce8-e335-4833-81b3-9c99d1897474") : configmap "kube-rbac-proxy" not found Sep 17 18:04:08.472310 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:08.472280 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:04:08.472387 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:08.472351 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls podName:99fc2ce8-e335-4833-81b3-9c99d1897474 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:16.472333772 +0000 UTC m=+81.482520901 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22vs2" (UID: "99fc2ce8-e335-4833-81b3-9c99d1897474") : secret "insights-runtime-extractor-tls" not found Sep 17 18:04:08.572820 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:08.572794 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6996l\" (UniqueName: \"kubernetes.io/projected/c30c2ea2-d35b-4ac0-b0c7-d4e059951efc-kube-api-access-6996l\") pod \"network-check-source-7cbc45fddc-pgpj7\" (UID: \"c30c2ea2-d35b-4ac0-b0c7-d4e059951efc\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-pgpj7" Sep 17 18:04:08.585723 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:08.585699 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6996l\" (UniqueName: \"kubernetes.io/projected/c30c2ea2-d35b-4ac0-b0c7-d4e059951efc-kube-api-access-6996l\") pod \"network-check-source-7cbc45fddc-pgpj7\" (UID: \"c30c2ea2-d35b-4ac0-b0c7-d4e059951efc\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-pgpj7" Sep 17 18:04:08.761990 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:08.761926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-pgpj7" Sep 17 18:04:09.038567 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:09.038482 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-pgpj7"] Sep 17 18:04:09.041464 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:09.041441 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30c2ea2_d35b_4ac0_b0c7_d4e059951efc.slice/crio-7d62d582f4cbf404a0f493b513884c18fee8bab5ec658cb637eca13768bdc924 WatchSource:0}: Error finding container 7d62d582f4cbf404a0f493b513884c18fee8bab5ec658cb637eca13768bdc924: Status 404 returned error can't find the container with id 7d62d582f4cbf404a0f493b513884c18fee8bab5ec658cb637eca13768bdc924 Sep 17 18:04:09.858688 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:09.858654 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-pgpj7" event={"ID":"c30c2ea2-d35b-4ac0-b0c7-d4e059951efc","Type":"ContainerStarted","Data":"5f7dbdb0b6c04efae2ec9e7e9a1dc5e11e198f6352795feebe9e96b5c2443f5e"} Sep 17 18:04:09.859030 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:09.858703 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-pgpj7" event={"ID":"c30c2ea2-d35b-4ac0-b0c7-d4e059951efc","Type":"ContainerStarted","Data":"7d62d582f4cbf404a0f493b513884c18fee8bab5ec658cb637eca13768bdc924"} Sep 17 18:04:09.878354 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:09.878310 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-pgpj7" podStartSLOduration=1.8782970049999999 podStartE2EDuration="1.878297005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:09.878275616 +0000 UTC m=+74.888462754" watchObservedRunningTime="2026-09-17 18:04:09.878297005 +0000 UTC m=+74.888484141" Sep 17 18:04:12.599881 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:12.599832 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:12.600303 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:12.599934 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:12.600303 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:12.599982 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:04:12.600303 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:12.600060 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs podName:8cc392b6-399b-4293-9c92-afc8dbf7d952 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:28.600044329 +0000 UTC m=+93.610231443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs") pod "router-default-64665745fb-n4rxl" (UID: "8cc392b6-399b-4293-9c92-afc8dbf7d952") : secret "router-metrics-certs-default" not found Sep 17 18:04:12.600303 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:12.600076 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle podName:8cc392b6-399b-4293-9c92-afc8dbf7d952 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:28.600069569 +0000 UTC m=+93.610256683 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle") pod "router-default-64665745fb-n4rxl" (UID: "8cc392b6-399b-4293-9c92-afc8dbf7d952") : configmap references non-existent config key: service-ca.crt Sep 17 18:04:13.855548 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:13.855495 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-fcmsg"] Sep 17 18:04:13.857519 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:13.857498 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" Sep 17 18:04:13.860368 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:13.860344 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 17 18:04:13.861296 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:13.861273 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 17 18:04:13.861413 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:13.861328 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 17 18:04:13.861413 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:13.861370 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4rp9z\"" Sep 17 18:04:13.861561 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:13.861424 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 17 18:04:13.871971 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:13.871943 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-fcmsg"] Sep 17 18:04:13.909841 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:13.909819 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6306140f-beff-41da-adbc-4bdac2b23eef-signing-key\") pod \"service-ca-55f6fc7b86-fcmsg\" (UID: \"6306140f-beff-41da-adbc-4bdac2b23eef\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" Sep 17 18:04:13.909969 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:13.909950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6306140f-beff-41da-adbc-4bdac2b23eef-signing-cabundle\") pod \"service-ca-55f6fc7b86-fcmsg\" (UID: \"6306140f-beff-41da-adbc-4bdac2b23eef\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" Sep 17 18:04:13.910051 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:13.910031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjf54\" (UniqueName: \"kubernetes.io/projected/6306140f-beff-41da-adbc-4bdac2b23eef-kube-api-access-vjf54\") pod \"service-ca-55f6fc7b86-fcmsg\" (UID: \"6306140f-beff-41da-adbc-4bdac2b23eef\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" Sep 17 18:04:14.010729 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:14.010699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6306140f-beff-41da-adbc-4bdac2b23eef-signing-cabundle\") pod \"service-ca-55f6fc7b86-fcmsg\" (UID: \"6306140f-beff-41da-adbc-4bdac2b23eef\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" Sep 17 18:04:14.010846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:14.010750 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjf54\" (UniqueName: \"kubernetes.io/projected/6306140f-beff-41da-adbc-4bdac2b23eef-kube-api-access-vjf54\") pod \"service-ca-55f6fc7b86-fcmsg\" (UID: \"6306140f-beff-41da-adbc-4bdac2b23eef\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" Sep 17 18:04:14.010846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:14.010768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6306140f-beff-41da-adbc-4bdac2b23eef-signing-key\") pod \"service-ca-55f6fc7b86-fcmsg\" (UID: \"6306140f-beff-41da-adbc-4bdac2b23eef\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" Sep 17 18:04:14.011351 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:14.011327 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6306140f-beff-41da-adbc-4bdac2b23eef-signing-cabundle\") pod \"service-ca-55f6fc7b86-fcmsg\" (UID: \"6306140f-beff-41da-adbc-4bdac2b23eef\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" Sep 17 18:04:14.013260 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:14.013241 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6306140f-beff-41da-adbc-4bdac2b23eef-signing-key\") pod \"service-ca-55f6fc7b86-fcmsg\" (UID: \"6306140f-beff-41da-adbc-4bdac2b23eef\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" Sep 17 18:04:14.020818 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:14.020789 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjf54\" (UniqueName: \"kubernetes.io/projected/6306140f-beff-41da-adbc-4bdac2b23eef-kube-api-access-vjf54\") pod \"service-ca-55f6fc7b86-fcmsg\" (UID: \"6306140f-beff-41da-adbc-4bdac2b23eef\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" Sep 17 18:04:14.167000 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:14.166954 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" Sep 17 18:04:14.445819 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:14.445793 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-fcmsg"] Sep 17 18:04:14.447947 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:14.447918 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6306140f_beff_41da_adbc_4bdac2b23eef.slice/crio-64667df15c0f9ce7e47ce163a0028b9e3ede6778d4dfdaf8b197cf18fa0a26cb WatchSource:0}: Error finding container 64667df15c0f9ce7e47ce163a0028b9e3ede6778d4dfdaf8b197cf18fa0a26cb: Status 404 returned error can't find the container with id 64667df15c0f9ce7e47ce163a0028b9e3ede6778d4dfdaf8b197cf18fa0a26cb Sep 17 18:04:14.875473 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:14.875427 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" event={"ID":"6306140f-beff-41da-adbc-4bdac2b23eef","Type":"ContainerStarted","Data":"64667df15c0f9ce7e47ce163a0028b9e3ede6778d4dfdaf8b197cf18fa0a26cb"} Sep 17 18:04:16.531053 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:16.530974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:16.531053 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:16.531014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:16.531428 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:16.531124 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:04:16.531428 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:16.531186 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls podName:99fc2ce8-e335-4833-81b3-9c99d1897474 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:32.531171217 +0000 UTC m=+97.541358331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22vs2" (UID: "99fc2ce8-e335-4833-81b3-9c99d1897474") : secret "insights-runtime-extractor-tls" not found Sep 17 18:04:16.531502 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:16.531470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99fc2ce8-e335-4833-81b3-9c99d1897474-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:16.883428 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:16.883398 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" event={"ID":"6306140f-beff-41da-adbc-4bdac2b23eef","Type":"ContainerStarted","Data":"3f9e19f6b93a08c6171e71bca4be4ac5a7bf6913e8058fd78cad537fb0900029"} Sep 17 18:04:16.905414 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:16.905361 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-fcmsg" podStartSLOduration=2.17388631 podStartE2EDuration="3.905343217s" totalImagesPullingTime="1.731456907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:16.903682023 +0000 UTC m=+81.913869158" watchObservedRunningTime="2026-09-17 18:04:16.905343217 +0000 UTC m=+81.915530354" Sep 17 18:04:28.616228 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:28.616181 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:28.616228 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:28.616255 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:28.616893 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:28.616866 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cc392b6-399b-4293-9c92-afc8dbf7d952-service-ca-bundle\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:28.619199 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:28.619173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc392b6-399b-4293-9c92-afc8dbf7d952-metrics-certs\") pod \"router-default-64665745fb-n4rxl\" (UID: \"8cc392b6-399b-4293-9c92-afc8dbf7d952\") " pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:28.900635 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:28.900561 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:29.191077 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:29.190897 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64665745fb-n4rxl"] Sep 17 18:04:29.193728 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:29.193703 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc392b6_399b_4293_9c92_afc8dbf7d952.slice/crio-11770ca649065baf210fdb383e7b926113951b44cf61a2eb9740184c441345e3 WatchSource:0}: Error finding container 11770ca649065baf210fdb383e7b926113951b44cf61a2eb9740184c441345e3: Status 404 returned error can't find the container with id 11770ca649065baf210fdb383e7b926113951b44cf61a2eb9740184c441345e3 Sep 17 18:04:29.926008 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:29.925969 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64665745fb-n4rxl" event={"ID":"8cc392b6-399b-4293-9c92-afc8dbf7d952","Type":"ContainerStarted","Data":"ea900121e5b5caad669d7b932c1d4edaad365b3f60c787ee0e355363a8099bcf"} Sep 17 18:04:29.926349 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:29.926024 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64665745fb-n4rxl" event={"ID":"8cc392b6-399b-4293-9c92-afc8dbf7d952","Type":"ContainerStarted","Data":"11770ca649065baf210fdb383e7b926113951b44cf61a2eb9740184c441345e3"} Sep 17 18:04:30.901769 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:30.901733 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:30.904484 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:30.904459 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:30.929796 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:30.929777 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:30.930948 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:30.930928 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64665745fb-n4rxl" Sep 17 18:04:30.987254 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:30.987194 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64665745fb-n4rxl" podStartSLOduration=34.987180074 podStartE2EDuration="34.987180074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:29.962598537 +0000 UTC m=+94.972785673" watchObservedRunningTime="2026-09-17 18:04:30.987180074 +0000 UTC m=+95.997367206" Sep 17 18:04:32.545031 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:32.544999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:32.547540 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:32.547506 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99fc2ce8-e335-4833-81b3-9c99d1897474-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22vs2\" (UID: \"99fc2ce8-e335-4833-81b3-9c99d1897474\") " pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:32.740964 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:32.740889 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-22vs2" Sep 17 18:04:32.746590 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:32.746567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:04:32.749018 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:32.748991 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8385b17-3d85-46f2-abbc-f92dd8cc9357-metrics-tls\") pod \"dns-default-9zglp\" (UID: \"b8385b17-3d85-46f2-abbc-f92dd8cc9357\") " pod="openshift-dns/dns-default-9zglp" Sep 17 18:04:32.800896 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:32.800864 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7dfk9\"" Sep 17 18:04:32.809508 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:32.809413 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9zglp" Sep 17 18:04:33.030193 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:33.030129 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-22vs2"] Sep 17 18:04:33.033228 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:33.033206 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99fc2ce8_e335_4833_81b3_9c99d1897474.slice/crio-e62a24e81c60a92e9a50560ef1f10fe9d3cf7342248b2120235879aca86cae19 WatchSource:0}: Error finding container e62a24e81c60a92e9a50560ef1f10fe9d3cf7342248b2120235879aca86cae19: Status 404 returned error can't find the container with id e62a24e81c60a92e9a50560ef1f10fe9d3cf7342248b2120235879aca86cae19 Sep 17 18:04:33.088323 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:33.088261 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9zglp"] Sep 17 18:04:33.092762 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:33.092741 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8385b17_3d85_46f2_abbc_f92dd8cc9357.slice/crio-824a4011b3c20b44a8cc3975260c4c364f57930837980c47e9b3e153f771e1d7 WatchSource:0}: Error finding container 824a4011b3c20b44a8cc3975260c4c364f57930837980c47e9b3e153f771e1d7: Status 404 returned error can't find the container with id 824a4011b3c20b44a8cc3975260c4c364f57930837980c47e9b3e153f771e1d7 Sep 17 18:04:33.938999 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:33.938927 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9zglp" event={"ID":"b8385b17-3d85-46f2-abbc-f92dd8cc9357","Type":"ContainerStarted","Data":"824a4011b3c20b44a8cc3975260c4c364f57930837980c47e9b3e153f771e1d7"} Sep 17 18:04:33.941003 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:33.940975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22vs2" event={"ID":"99fc2ce8-e335-4833-81b3-9c99d1897474","Type":"ContainerStarted","Data":"d9046c06979649420c9bba38864f8b557d66f98ea39120353b302eecae928532"} Sep 17 18:04:33.941129 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:33.941023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22vs2" event={"ID":"99fc2ce8-e335-4833-81b3-9c99d1897474","Type":"ContainerStarted","Data":"254113c1fb900fbcae81818f8d8ca6fbeb795ed35867fcf2ff8e6ed0db12d15c"} Sep 17 18:04:33.941129 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:33.941044 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22vs2" event={"ID":"99fc2ce8-e335-4833-81b3-9c99d1897474","Type":"ContainerStarted","Data":"e62a24e81c60a92e9a50560ef1f10fe9d3cf7342248b2120235879aca86cae19"} Sep 17 18:04:35.950058 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:35.950023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9zglp" event={"ID":"b8385b17-3d85-46f2-abbc-f92dd8cc9357","Type":"ContainerStarted","Data":"f387d141363c2b230167719578c2a2c2724ddeacd84f3d6e121f6cbfed553c5a"} Sep 17 18:04:35.950470 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:35.950084 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9zglp" event={"ID":"b8385b17-3d85-46f2-abbc-f92dd8cc9357","Type":"ContainerStarted","Data":"75a232dc886849b44253467a5c440c8305703b6885028ccdffeae0acd61562f1"} Sep 17 18:04:35.950470 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:35.950113 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9zglp" Sep 17 18:04:35.952284 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:35.952255 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22vs2" event={"ID":"99fc2ce8-e335-4833-81b3-9c99d1897474","Type":"ContainerStarted","Data":"2df76fe39240ea181cbbfb440ce7800459287ba19bd3d39279f332d6ff38277d"} Sep 17 18:04:35.980069 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:35.980026 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9zglp" podStartSLOduration=65.724304385 podStartE2EDuration="1m7.98001272s" totalImagesPullingTime="2.255708335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:35.97948683 +0000 UTC m=+100.989673966" watchObservedRunningTime="2026-09-17 18:04:35.98001272 +0000 UTC m=+100.990199861" Sep 17 18:04:36.000323 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.000297 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4"] Sep 17 18:04:36.003118 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.003104 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4" Sep 17 18:04:36.006459 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.006441 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2jvgz\"" Sep 17 18:04:36.006845 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.006829 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 17 18:04:36.006912 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.006832 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 17 18:04:36.007227 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.007207 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 17 18:04:36.007296 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.007268 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 17 18:04:36.027226 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.027208 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4"] Sep 17 18:04:36.027973 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.027935 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-22vs2" podStartSLOduration=33.815238861 podStartE2EDuration="36.027922362s" totalImagesPullingTime="2.212683501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:04:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:36.026573536 +0000 UTC m=+101.036760675" watchObservedRunningTime="2026-09-17 18:04:36.027922362 +0000 UTC m=+101.038109503" Sep 17 18:04:36.071227 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.071199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd9cd34b-4f69-4181-b818-14459a03d789-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6589f858c6-5qfs4\" (UID: \"fd9cd34b-4f69-4181-b818-14459a03d789\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4" Sep 17 18:04:36.071321 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.071258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6rz\" (UniqueName: \"kubernetes.io/projected/fd9cd34b-4f69-4181-b818-14459a03d789-kube-api-access-cf6rz\") pod \"managed-serviceaccount-addon-agent-6589f858c6-5qfs4\" (UID: \"fd9cd34b-4f69-4181-b818-14459a03d789\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4" Sep 17 18:04:36.171754 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.171716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd9cd34b-4f69-4181-b818-14459a03d789-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6589f858c6-5qfs4\" (UID: \"fd9cd34b-4f69-4181-b818-14459a03d789\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4" Sep 17 18:04:36.171861 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.171801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6rz\" (UniqueName: \"kubernetes.io/projected/fd9cd34b-4f69-4181-b818-14459a03d789-kube-api-access-cf6rz\") pod \"managed-serviceaccount-addon-agent-6589f858c6-5qfs4\" (UID: \"fd9cd34b-4f69-4181-b818-14459a03d789\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4" Sep 17 18:04:36.174211 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.174188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd9cd34b-4f69-4181-b818-14459a03d789-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6589f858c6-5qfs4\" (UID: \"fd9cd34b-4f69-4181-b818-14459a03d789\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4" Sep 17 18:04:36.200879 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.200830 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6rz\" (UniqueName: \"kubernetes.io/projected/fd9cd34b-4f69-4181-b818-14459a03d789-kube-api-access-cf6rz\") pod \"managed-serviceaccount-addon-agent-6589f858c6-5qfs4\" (UID: \"fd9cd34b-4f69-4181-b818-14459a03d789\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4" Sep 17 18:04:36.251321 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.251300 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-95jxg"] Sep 17 18:04:36.254326 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.254309 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-95jxg" Sep 17 18:04:36.257318 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.257300 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-p4mpn\"" Sep 17 18:04:36.259958 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.259938 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 17 18:04:36.309377 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.309359 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-95jxg"] Sep 17 18:04:36.323915 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.323899 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4" Sep 17 18:04:36.342093 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.342074 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d97cbd56c-r9w88"] Sep 17 18:04:36.345161 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.345146 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.351133 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.351101 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 18:04:36.352977 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.352959 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 18:04:36.353136 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.353118 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 18:04:36.355576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.355561 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kgd2r\"" Sep 17 18:04:36.375016 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.374080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7e3b9c9b-e1e6-46ab-879e-b7e3b35538af-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-95jxg\" (UID: \"7e3b9c9b-e1e6-46ab-879e-b7e3b35538af\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-95jxg" Sep 17 18:04:36.375016 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.374945 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d97cbd56c-r9w88"] Sep 17 18:04:36.386844 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.386823 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 18:04:36.474627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.474566 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-trusted-ca\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.474627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.474604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-registry-tls\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.474627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.474624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-installation-pull-secrets\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.474847 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.474640 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5djv\" (UniqueName: \"kubernetes.io/projected/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-kube-api-access-g5djv\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.474847 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.474729 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7e3b9c9b-e1e6-46ab-879e-b7e3b35538af-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-95jxg\" (UID: \"7e3b9c9b-e1e6-46ab-879e-b7e3b35538af\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-95jxg" Sep 17 18:04:36.474847 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.474765 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-image-registry-private-configuration\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.474847 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.474801 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-ca-trust-extracted\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.474847 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.474830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-bound-sa-token\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.475051 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.474902 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-registry-certificates\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.477072 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.477052 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7e3b9c9b-e1e6-46ab-879e-b7e3b35538af-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-95jxg\" (UID: \"7e3b9c9b-e1e6-46ab-879e-b7e3b35538af\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-95jxg" Sep 17 18:04:36.563035 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.563014 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-95jxg" Sep 17 18:04:36.575901 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.575876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-bound-sa-token\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.575989 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.575940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-registry-certificates\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.575989 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.575984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-trusted-ca\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.576098 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.576010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-registry-tls\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.576098 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.576037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-installation-pull-secrets\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.576098 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.576061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5djv\" (UniqueName: \"kubernetes.io/projected/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-kube-api-access-g5djv\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.576273 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.576110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-image-registry-private-configuration\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.576273 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.576141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-ca-trust-extracted\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.576508 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.576485 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-ca-trust-extracted\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.576917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.576890 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-registry-certificates\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.577278 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.577239 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-trusted-ca\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.579012 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.578992 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-registry-tls\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.579485 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.579265 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-image-registry-private-configuration\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.579916 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.579889 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-installation-pull-secrets\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.598275 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.598252 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-bound-sa-token\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.598540 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.598507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5djv\" (UniqueName: \"kubernetes.io/projected/a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43-kube-api-access-g5djv\") pod \"image-registry-d97cbd56c-r9w88\" (UID: \"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43\") " pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.616182 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.616156 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4"] Sep 17 18:04:36.653975 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.653951 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:36.844333 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.844309 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-95jxg"] Sep 17 18:04:36.848337 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:36.848315 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3b9c9b_e1e6_46ab_879e_b7e3b35538af.slice/crio-59580cf5337c2d87b7851bae228cc9869d98f6fe35c0a35734ab0abcc5c52318 WatchSource:0}: Error finding container 59580cf5337c2d87b7851bae228cc9869d98f6fe35c0a35734ab0abcc5c52318: Status 404 returned error can't find the container with id 59580cf5337c2d87b7851bae228cc9869d98f6fe35c0a35734ab0abcc5c52318 Sep 17 18:04:36.954455 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.954432 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d97cbd56c-r9w88"] Sep 17 18:04:36.957555 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.957506 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4" event={"ID":"fd9cd34b-4f69-4181-b818-14459a03d789","Type":"ContainerStarted","Data":"b007cb2d4181d4e9af20a739c51577d05480460da5248c48ed28e9da6a507dd0"} Sep 17 18:04:36.957874 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:36.957845 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d962b4_8f76_4dc7_be4a_e9ed7aa53a43.slice/crio-9b38eda29b7f051dceb647b9326fa911a7963c62b36b5e53b80d68d4eb4e49ef WatchSource:0}: Error finding container 9b38eda29b7f051dceb647b9326fa911a7963c62b36b5e53b80d68d4eb4e49ef: Status 404 returned error can't find the container with id 9b38eda29b7f051dceb647b9326fa911a7963c62b36b5e53b80d68d4eb4e49ef Sep 17 18:04:36.958858 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:36.958836 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-95jxg" event={"ID":"7e3b9c9b-e1e6-46ab-879e-b7e3b35538af","Type":"ContainerStarted","Data":"59580cf5337c2d87b7851bae228cc9869d98f6fe35c0a35734ab0abcc5c52318"} Sep 17 18:04:37.854825 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:37.854794 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b4sz9" Sep 17 18:04:37.967418 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:37.966663 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" event={"ID":"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43","Type":"ContainerStarted","Data":"c0aab173c69d56aa446bb3a456f10cafb05c6d38bf3a1cff1e34697cfd84736b"} Sep 17 18:04:37.967418 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:37.966722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" event={"ID":"a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43","Type":"ContainerStarted","Data":"9b38eda29b7f051dceb647b9326fa911a7963c62b36b5e53b80d68d4eb4e49ef"} Sep 17 18:04:37.967418 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:37.967350 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:04:38.001307 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:38.001208 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" podStartSLOduration=2.00118645 podStartE2EDuration="2.00118645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:37.999924873 +0000 UTC m=+103.010112011" watchObservedRunningTime="2026-09-17 18:04:38.00118645 +0000 UTC m=+103.011373587" Sep 17 18:04:39.975268 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:39.975236 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-95jxg" event={"ID":"7e3b9c9b-e1e6-46ab-879e-b7e3b35538af","Type":"ContainerStarted","Data":"38208cf647ef32abc16fc37504ae0491ac318299cc3252394f7a77531d9ea637"} Sep 17 18:04:39.975601 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:39.975412 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-95jxg" Sep 17 18:04:39.979976 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:39.979955 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-95jxg" Sep 17 18:04:39.999116 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:39.999075 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-95jxg" podStartSLOduration=1.020170376 podStartE2EDuration="3.99906146s" totalImagesPullingTime="2.978891084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:39.998488805 +0000 UTC m=+105.008675942" watchObservedRunningTime="2026-09-17 18:04:39.99906146 +0000 UTC m=+105.009248596" Sep 17 18:04:40.368717 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.368687 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf"] Sep 17 18:04:40.388784 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.388751 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf"] Sep 17 18:04:40.388911 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.388866 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.391622 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.391597 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 17 18:04:40.391802 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.391782 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 18:04:40.392689 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.392492 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 17 18:04:40.392689 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.392589 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ps6r4\"" Sep 17 18:04:40.392689 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.392621 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 18:04:40.392893 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.392696 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 18:04:40.510206 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.510147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/feceb9c7-986d-4b94-b85d-4aa1f42d1d7a-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-lm6lf\" (UID: \"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.510206 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.510180 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjd7x\" (UniqueName: \"kubernetes.io/projected/feceb9c7-986d-4b94-b85d-4aa1f42d1d7a-kube-api-access-gjd7x\") pod \"prometheus-operator-7bd5d56d6-lm6lf\" (UID: \"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.510206 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.510199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feceb9c7-986d-4b94-b85d-4aa1f42d1d7a-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-lm6lf\" (UID: \"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.510380 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.510272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/feceb9c7-986d-4b94-b85d-4aa1f42d1d7a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-lm6lf\" (UID: \"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.611146 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.611126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/feceb9c7-986d-4b94-b85d-4aa1f42d1d7a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-lm6lf\" (UID: \"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.611255 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.611167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/feceb9c7-986d-4b94-b85d-4aa1f42d1d7a-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-lm6lf\" (UID: \"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.611255 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.611189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjd7x\" (UniqueName: \"kubernetes.io/projected/feceb9c7-986d-4b94-b85d-4aa1f42d1d7a-kube-api-access-gjd7x\") pod \"prometheus-operator-7bd5d56d6-lm6lf\" (UID: \"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.611255 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.611207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feceb9c7-986d-4b94-b85d-4aa1f42d1d7a-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-lm6lf\" (UID: \"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.611770 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.611751 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feceb9c7-986d-4b94-b85d-4aa1f42d1d7a-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-lm6lf\" (UID: \"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.613611 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.613593 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/feceb9c7-986d-4b94-b85d-4aa1f42d1d7a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-lm6lf\" (UID: \"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.613797 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.613775 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/feceb9c7-986d-4b94-b85d-4aa1f42d1d7a-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-lm6lf\" (UID: \"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.620384 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.620358 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjd7x\" (UniqueName: \"kubernetes.io/projected/feceb9c7-986d-4b94-b85d-4aa1f42d1d7a-kube-api-access-gjd7x\") pod \"prometheus-operator-7bd5d56d6-lm6lf\" (UID: \"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.699300 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.699281 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" Sep 17 18:04:40.974507 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.974355 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf"] Sep 17 18:04:40.977024 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:40.977001 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeceb9c7_986d_4b94_b85d_4aa1f42d1d7a.slice/crio-92c88c0fbc22d457d1748d7cc26f6c44d8514e11b65b2cae63fd7925269284f4 WatchSource:0}: Error finding container 92c88c0fbc22d457d1748d7cc26f6c44d8514e11b65b2cae63fd7925269284f4: Status 404 returned error can't find the container with id 92c88c0fbc22d457d1748d7cc26f6c44d8514e11b65b2cae63fd7925269284f4 Sep 17 18:04:40.981091 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:40.980986 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4" event={"ID":"fd9cd34b-4f69-4181-b818-14459a03d789","Type":"ContainerStarted","Data":"3430e88248a1972518ca3b8641513773c96a1a459f4d4ec85e5494e0793e4fcd"} Sep 17 18:04:41.013721 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:41.013678 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6589f858c6-5qfs4" podStartSLOduration=2.389813662 podStartE2EDuration="6.013662939s" totalImagesPullingTime="3.623849277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:41.012223593 +0000 UTC m=+106.022410730" watchObservedRunningTime="2026-09-17 18:04:41.013662939 +0000 UTC m=+106.023850075" Sep 17 18:04:41.985485 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:41.985443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" event={"ID":"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a","Type":"ContainerStarted","Data":"92c88c0fbc22d457d1748d7cc26f6c44d8514e11b65b2cae63fd7925269284f4"} Sep 17 18:04:42.990851 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:42.990813 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" event={"ID":"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a","Type":"ContainerStarted","Data":"15b73f25dd2009e866d2b6bf370f67065cd5c89c4858db8edcdaad21e79eb827"} Sep 17 18:04:42.991231 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:42.990873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" event={"ID":"feceb9c7-986d-4b94-b85d-4aa1f42d1d7a","Type":"ContainerStarted","Data":"e2b2cb8e903d4af946f1b3bd4a859ce15a82e7d92d06bccf20e3df157a726cf0"} Sep 17 18:04:45.166052 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.165997 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lm6lf" podStartSLOduration=3.832488567 podStartE2EDuration="5.165976046s" totalImagesPullingTime="1.333487479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:43.020909903 +0000 UTC m=+108.031097039" watchObservedRunningTime="2026-09-17 18:04:45.165976046 +0000 UTC m=+110.176163184" Sep 17 18:04:45.166859 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.166843 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-slq6v"] Sep 17 18:04:45.170381 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.170357 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.173881 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.173859 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 18:04:45.174048 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.173931 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 18:04:45.175051 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.175032 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 18:04:45.175387 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.175372 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-27csq\"" Sep 17 18:04:45.176293 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.176275 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cf6cd8f45-psn52"] Sep 17 18:04:45.178571 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.178480 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.182275 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.182254 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ppsr6\"" Sep 17 18:04:45.182755 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.182738 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 17 18:04:45.183261 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.183243 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 17 18:04:45.186211 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.186127 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 17 18:04:45.186543 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.186507 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 17 18:04:45.186857 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.186838 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 17 18:04:45.187127 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.187107 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 17 18:04:45.189295 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.189277 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 17 18:04:45.190177 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.190162 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 17 18:04:45.199974 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.199949 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf6cd8f45-psn52"] Sep 17 18:04:45.246148 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.246115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-accelerators-collector-config\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.246259 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.246167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/df993f22-21e2-4efb-9fe8-9328204d734d-root\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.246259 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.246208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.246259 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.246250 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ff6x\" (UniqueName: \"kubernetes.io/projected/df993f22-21e2-4efb-9fe8-9328204d734d-kube-api-access-6ff6x\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.246404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.246270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df993f22-21e2-4efb-9fe8-9328204d734d-sys\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.246404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.246294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df993f22-21e2-4efb-9fe8-9328204d734d-metrics-client-ca\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.246404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.246351 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-wtmp\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.246404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.246388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-tls\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.246614 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.246409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-textfile\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.347382 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347362 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/df993f22-21e2-4efb-9fe8-9328204d734d-root\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.347499 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347389 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.347499 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq6m7\" (UniqueName: \"kubernetes.io/projected/f17a0b28-d401-41e9-a051-4911f38043d9-kube-api-access-mq6m7\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.347499 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ff6x\" (UniqueName: \"kubernetes.io/projected/df993f22-21e2-4efb-9fe8-9328204d734d-kube-api-access-6ff6x\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.347499 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df993f22-21e2-4efb-9fe8-9328204d734d-sys\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.347499 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df993f22-21e2-4efb-9fe8-9328204d734d-metrics-client-ca\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.347777 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347506 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/df993f22-21e2-4efb-9fe8-9328204d734d-root\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.347777 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347514 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-wtmp\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.347777 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347639 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df993f22-21e2-4efb-9fe8-9328204d734d-sys\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.347777 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347650 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-oauth-serving-cert\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.347777 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347689 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-wtmp\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.347777 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-tls\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.347777 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f17a0b28-d401-41e9-a051-4911f38043d9-console-oauth-config\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.348104 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347786 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-textfile\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.348104 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-console-config\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.348104 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-service-ca\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.348104 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347879 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-trusted-ca-bundle\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.348104 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-accelerators-collector-config\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.348104 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.347991 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f17a0b28-d401-41e9-a051-4911f38043d9-console-serving-cert\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.348371 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.348210 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-textfile\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.348710 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.348680 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-accelerators-collector-config\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.348936 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.348909 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df993f22-21e2-4efb-9fe8-9328204d734d-metrics-client-ca\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.350101 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.350073 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-tls\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.350209 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.350192 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df993f22-21e2-4efb-9fe8-9328204d734d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.357284 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.357258 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ff6x\" (UniqueName: \"kubernetes.io/projected/df993f22-21e2-4efb-9fe8-9328204d734d-kube-api-access-6ff6x\") pod \"node-exporter-slq6v\" (UID: \"df993f22-21e2-4efb-9fe8-9328204d734d\") " pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.448412 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.448346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f17a0b28-d401-41e9-a051-4911f38043d9-console-serving-cert\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.448412 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.448376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq6m7\" (UniqueName: \"kubernetes.io/projected/f17a0b28-d401-41e9-a051-4911f38043d9-kube-api-access-mq6m7\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.448596 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.448416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-oauth-serving-cert\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.448596 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.448445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f17a0b28-d401-41e9-a051-4911f38043d9-console-oauth-config\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.448596 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.448476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-console-config\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.448596 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.448506 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-service-ca\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.448596 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.448554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-trusted-ca-bundle\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.449175 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.449151 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-oauth-serving-cert\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.450223 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.449364 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-trusted-ca-bundle\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.450223 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.450132 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-service-ca\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.450223 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.450212 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-console-config\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.451926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.451907 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f17a0b28-d401-41e9-a051-4911f38043d9-console-oauth-config\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.452348 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.452329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f17a0b28-d401-41e9-a051-4911f38043d9-console-serving-cert\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.458287 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.458261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq6m7\" (UniqueName: \"kubernetes.io/projected/f17a0b28-d401-41e9-a051-4911f38043d9-kube-api-access-mq6m7\") pod \"console-6cf6cd8f45-psn52\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.481284 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.481261 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slq6v" Sep 17 18:04:45.489639 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.489622 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:45.491396 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:45.491369 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf993f22_21e2_4efb_9fe8_9328204d734d.slice/crio-aa356fc53ec97ae48ca723f938afdd6b5d2900d0da6c9d6b5abd9d9c3c7d7d1c WatchSource:0}: Error finding container aa356fc53ec97ae48ca723f938afdd6b5d2900d0da6c9d6b5abd9d9c3c7d7d1c: Status 404 returned error can't find the container with id aa356fc53ec97ae48ca723f938afdd6b5d2900d0da6c9d6b5abd9d9c3c7d7d1c Sep 17 18:04:45.806831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.806811 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf6cd8f45-psn52"] Sep 17 18:04:45.809018 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:45.808993 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17a0b28_d401_41e9_a051_4911f38043d9.slice/crio-2a959c458a48af43131a143963f01dc0dced861ee192aaac51f6e72b37758dfb WatchSource:0}: Error finding container 2a959c458a48af43131a143963f01dc0dced861ee192aaac51f6e72b37758dfb: Status 404 returned error can't find the container with id 2a959c458a48af43131a143963f01dc0dced861ee192aaac51f6e72b37758dfb Sep 17 18:04:45.961456 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:45.961432 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9zglp" Sep 17 18:04:46.006564 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:46.005634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf6cd8f45-psn52" event={"ID":"f17a0b28-d401-41e9-a051-4911f38043d9","Type":"ContainerStarted","Data":"2a959c458a48af43131a143963f01dc0dced861ee192aaac51f6e72b37758dfb"} Sep 17 18:04:46.007147 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:46.007120 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slq6v" event={"ID":"df993f22-21e2-4efb-9fe8-9328204d734d","Type":"ContainerStarted","Data":"aa356fc53ec97ae48ca723f938afdd6b5d2900d0da6c9d6b5abd9d9c3c7d7d1c"} Sep 17 18:04:47.013838 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:47.013804 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slq6v" event={"ID":"df993f22-21e2-4efb-9fe8-9328204d734d","Type":"ContainerStarted","Data":"26dab939745e9719ad5d014d012001f6e052a14f37399b97cfe128dee785c7fb"} Sep 17 18:04:48.025595 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:48.025554 2564 generic.go:356] "Generic (PLEG): container finished" podID="df993f22-21e2-4efb-9fe8-9328204d734d" containerID="26dab939745e9719ad5d014d012001f6e052a14f37399b97cfe128dee785c7fb" exitCode=0 Sep 17 18:04:48.026032 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:48.025634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slq6v" event={"ID":"df993f22-21e2-4efb-9fe8-9328204d734d","Type":"ContainerDied","Data":"26dab939745e9719ad5d014d012001f6e052a14f37399b97cfe128dee785c7fb"} Sep 17 18:04:49.056572 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.056508 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7b67466fcd-h7hn9"] Sep 17 18:04:49.060521 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.060484 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.064930 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.064909 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 17 18:04:49.065172 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.065154 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1sd90t9jf68e4\"" Sep 17 18:04:49.065422 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.065404 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 17 18:04:49.065565 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.065455 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qsrdf\"" Sep 17 18:04:49.065565 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.065459 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 17 18:04:49.065761 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.065744 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 17 18:04:49.066069 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.066052 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 17 18:04:49.096031 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.096005 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b67466fcd-h7hn9"] Sep 17 18:04:49.182147 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.182114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51a10561-f1f2-4157-a74e-8ff241d6a9e2-metrics-client-ca\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.182284 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.182166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.182284 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.182200 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.182284 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.182237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-tls\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.182284 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.182265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.182447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.182353 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-grpc-tls\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.182447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.182400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.182447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.182436 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6t46\" (UniqueName: \"kubernetes.io/projected/51a10561-f1f2-4157-a74e-8ff241d6a9e2-kube-api-access-d6t46\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.283753 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.283716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51a10561-f1f2-4157-a74e-8ff241d6a9e2-metrics-client-ca\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.283917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.283776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.283917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.283811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.283917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.283848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-tls\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.283917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.283876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.284136 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.283921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-grpc-tls\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.284136 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.283947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.284136 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.283972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6t46\" (UniqueName: \"kubernetes.io/projected/51a10561-f1f2-4157-a74e-8ff241d6a9e2-kube-api-access-d6t46\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.284603 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.284555 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51a10561-f1f2-4157-a74e-8ff241d6a9e2-metrics-client-ca\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.287726 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.287414 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.287726 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.287437 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-tls\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.287726 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.287554 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.287950 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.287884 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-grpc-tls\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.288017 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.287996 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.288070 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.288045 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/51a10561-f1f2-4157-a74e-8ff241d6a9e2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.292919 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.292896 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6t46\" (UniqueName: \"kubernetes.io/projected/51a10561-f1f2-4157-a74e-8ff241d6a9e2-kube-api-access-d6t46\") pod \"thanos-querier-7b67466fcd-h7hn9\" (UID: \"51a10561-f1f2-4157-a74e-8ff241d6a9e2\") " pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.371960 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.371924 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:49.816963 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.816919 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-gq22m"] Sep 17 18:04:49.821071 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.821042 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" Sep 17 18:04:49.829088 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.829069 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 17 18:04:49.829352 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.829336 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mpdkd\"" Sep 17 18:04:49.841314 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.840362 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-gq22m"] Sep 17 18:04:49.989753 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.989721 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4a5e89b-2526-495d-8321-fc7825a6e74e-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-gq22m\" (UID: \"a4a5e89b-2526-495d-8321-fc7825a6e74e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" Sep 17 18:04:49.989753 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:49.989752 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xcqs\" (UniqueName: \"kubernetes.io/projected/a4a5e89b-2526-495d-8321-fc7825a6e74e-kube-api-access-6xcqs\") pod \"monitoring-plugin-5984459ff8-gq22m\" (UID: \"a4a5e89b-2526-495d-8321-fc7825a6e74e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" Sep 17 18:04:50.035294 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.035255 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf6cd8f45-psn52" event={"ID":"f17a0b28-d401-41e9-a051-4911f38043d9","Type":"ContainerStarted","Data":"7339d921d41a887658206104d48f24b8af1d14880af84d127355866e3a285a8a"} Sep 17 18:04:50.037753 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.037726 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slq6v" event={"ID":"df993f22-21e2-4efb-9fe8-9328204d734d","Type":"ContainerStarted","Data":"d717004e0370220c1c103449dd99f98d16592aec3c768622963cbb2cfa88cf93"} Sep 17 18:04:50.037895 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.037786 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slq6v" event={"ID":"df993f22-21e2-4efb-9fe8-9328204d734d","Type":"ContainerStarted","Data":"b04c01e2ffeb4935f9160d664c43894bc076a8869d9300e95c761fd5945d89bb"} Sep 17 18:04:50.059438 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.059367 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cf6cd8f45-psn52" podStartSLOduration=1.006292823 podStartE2EDuration="5.059353767s" totalImagesPullingTime="4.053060944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:50.057769527 +0000 UTC m=+115.067956663" watchObservedRunningTime="2026-09-17 18:04:50.059353767 +0000 UTC m=+115.069540902" Sep 17 18:04:50.084023 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.083948 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b67466fcd-h7hn9"] Sep 17 18:04:50.085921 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:50.085902 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a10561_f1f2_4157_a74e_8ff241d6a9e2.slice/crio-68b176901c8a8d18fb41702c2ebcea58e015e670de9a70d8121832f02e229fc8 WatchSource:0}: Error finding container 68b176901c8a8d18fb41702c2ebcea58e015e670de9a70d8121832f02e229fc8: Status 404 returned error can't find the container with id 68b176901c8a8d18fb41702c2ebcea58e015e670de9a70d8121832f02e229fc8 Sep 17 18:04:50.090700 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.090672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4a5e89b-2526-495d-8321-fc7825a6e74e-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-gq22m\" (UID: \"a4a5e89b-2526-495d-8321-fc7825a6e74e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" Sep 17 18:04:50.090783 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.090709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xcqs\" (UniqueName: \"kubernetes.io/projected/a4a5e89b-2526-495d-8321-fc7825a6e74e-kube-api-access-6xcqs\") pod \"monitoring-plugin-5984459ff8-gq22m\" (UID: \"a4a5e89b-2526-495d-8321-fc7825a6e74e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" Sep 17 18:04:50.090838 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:50.090820 2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 17 18:04:50.090898 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:04:50.090887 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4a5e89b-2526-495d-8321-fc7825a6e74e-monitoring-plugin-cert podName:a4a5e89b-2526-495d-8321-fc7825a6e74e nodeName:}" failed. No retries permitted until 2026-09-17 18:04:50.590869753 +0000 UTC m=+115.601056884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a4a5e89b-2526-495d-8321-fc7825a6e74e-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-gq22m" (UID: "a4a5e89b-2526-495d-8321-fc7825a6e74e") : secret "monitoring-plugin-cert" not found Sep 17 18:04:50.091483 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.091451 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-slq6v" podStartSLOduration=3.652816502 podStartE2EDuration="5.091438382s" totalImagesPullingTime="1.358691452s" totalInitContainerRuntime="79.930428ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:50.088610927 +0000 UTC m=+115.098798063" watchObservedRunningTime="2026-09-17 18:04:50.091438382 +0000 UTC m=+115.101625519" Sep 17 18:04:50.104373 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.104351 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xcqs\" (UniqueName: \"kubernetes.io/projected/a4a5e89b-2526-495d-8321-fc7825a6e74e-kube-api-access-6xcqs\") pod \"monitoring-plugin-5984459ff8-gq22m\" (UID: \"a4a5e89b-2526-495d-8321-fc7825a6e74e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" Sep 17 18:04:50.443432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.443327 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-68b846668c-m2vs5"] Sep 17 18:04:50.451456 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.451435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.455441 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.455417 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 17 18:04:50.455441 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.455441 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-5zrf6\"" Sep 17 18:04:50.455759 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.455729 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 17 18:04:50.455850 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.455770 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 17 18:04:50.456017 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.455985 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 17 18:04:50.459478 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.459460 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 17 18:04:50.462540 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.462512 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 17 18:04:50.468985 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.468967 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68b846668c-m2vs5"] Sep 17 18:04:50.595321 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.595295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26kp\" (UniqueName: \"kubernetes.io/projected/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-kube-api-access-g26kp\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.595405 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.595325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-federate-client-tls\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.595405 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.595344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-telemeter-client-tls\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.595405 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.595366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4a5e89b-2526-495d-8321-fc7825a6e74e-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-gq22m\" (UID: \"a4a5e89b-2526-495d-8321-fc7825a6e74e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" Sep 17 18:04:50.595502 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.595442 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-metrics-client-ca\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.595502 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.595482 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.595590 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.595503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-secret-telemeter-client\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.595590 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.595565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.595648 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.595590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-serving-certs-ca-bundle\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.597901 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.597873 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4a5e89b-2526-495d-8321-fc7825a6e74e-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-gq22m\" (UID: \"a4a5e89b-2526-495d-8321-fc7825a6e74e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" Sep 17 18:04:50.696161 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.696089 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.696161 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.696124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-serving-certs-ca-bundle\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.696161 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.696155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26kp\" (UniqueName: \"kubernetes.io/projected/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-kube-api-access-g26kp\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.696367 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.696181 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-federate-client-tls\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.696367 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.696210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-telemeter-client-tls\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.696367 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.696269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-metrics-client-ca\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.696367 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.696297 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.696367 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.696323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-secret-telemeter-client\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.697014 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.696968 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-serving-certs-ca-bundle\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.697131 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.697112 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-metrics-client-ca\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.697478 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.697277 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.699574 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.699553 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-telemeter-client-tls\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.699802 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.699786 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.699846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.699825 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-secret-telemeter-client\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.699880 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.699831 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-federate-client-tls\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.705869 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.705845 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26kp\" (UniqueName: \"kubernetes.io/projected/ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691-kube-api-access-g26kp\") pod \"telemeter-client-68b846668c-m2vs5\" (UID: \"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691\") " pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:50.742371 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.742211 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" Sep 17 18:04:50.761382 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:50.761361 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" Sep 17 18:04:51.020729 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:51.020578 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-gq22m"] Sep 17 18:04:51.025606 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:51.025579 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a5e89b_2526_495d_8321_fc7825a6e74e.slice/crio-1b6d51899a70a462869a987418ebd22115af1551c67a3a01d7c01578c6cd0b4e WatchSource:0}: Error finding container 1b6d51899a70a462869a987418ebd22115af1551c67a3a01d7c01578c6cd0b4e: Status 404 returned error can't find the container with id 1b6d51899a70a462869a987418ebd22115af1551c67a3a01d7c01578c6cd0b4e Sep 17 18:04:51.043506 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:51.043473 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" event={"ID":"a4a5e89b-2526-495d-8321-fc7825a6e74e","Type":"ContainerStarted","Data":"1b6d51899a70a462869a987418ebd22115af1551c67a3a01d7c01578c6cd0b4e"} Sep 17 18:04:51.044979 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:51.044926 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" event={"ID":"51a10561-f1f2-4157-a74e-8ff241d6a9e2","Type":"ContainerStarted","Data":"68b176901c8a8d18fb41702c2ebcea58e015e670de9a70d8121832f02e229fc8"} Sep 17 18:04:51.053806 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:51.053781 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68b846668c-m2vs5"] Sep 17 18:04:51.749044 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:51.749011 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cf6cd8f45-psn52"] Sep 17 18:04:52.049972 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:52.049905 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" event={"ID":"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691","Type":"ContainerStarted","Data":"4b556326da4c38735e718a54a08bd50c721a2a6bac8f26a2fb1b8b6504106900"} Sep 17 18:04:53.061389 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:53.061349 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" event={"ID":"51a10561-f1f2-4157-a74e-8ff241d6a9e2","Type":"ContainerStarted","Data":"8069ad0d48b9c5115653f0863766d0f59e8ea67f9a028a203c3c5ee377be473a"} Sep 17 18:04:53.061796 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:53.061465 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" event={"ID":"51a10561-f1f2-4157-a74e-8ff241d6a9e2","Type":"ContainerStarted","Data":"731683974cba366747a204264b2a928b0068bc93895bb74a9e48433fecb521f1"} Sep 17 18:04:53.061796 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:53.061509 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" event={"ID":"51a10561-f1f2-4157-a74e-8ff241d6a9e2","Type":"ContainerStarted","Data":"bcd10a471342bb406f6ceb49d980f3274421adc95eff76d62667e062124d8ed0"} Sep 17 18:04:54.066303 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:54.066273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" event={"ID":"a4a5e89b-2526-495d-8321-fc7825a6e74e","Type":"ContainerStarted","Data":"158d73ac71c675d78aa74e60eca3b87d4410121d90669229bc1fa81c19d6bc4d"} Sep 17 18:04:54.066739 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:54.066457 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" Sep 17 18:04:54.069658 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:54.069634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" event={"ID":"51a10561-f1f2-4157-a74e-8ff241d6a9e2","Type":"ContainerStarted","Data":"b52cfb2e71154900937815c6057b85849cb7b7f5d31054566f513b0a23aaa560"} Sep 17 18:04:54.069793 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:54.069693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" event={"ID":"51a10561-f1f2-4157-a74e-8ff241d6a9e2","Type":"ContainerStarted","Data":"f36a78ce25c9543386fefeb47ea86444440ea0b4bd5638d9d2bc57af906635c8"} Sep 17 18:04:54.069793 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:54.069724 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" event={"ID":"51a10561-f1f2-4157-a74e-8ff241d6a9e2","Type":"ContainerStarted","Data":"add60a0ab057bb61bbe0a51529df178c4e984c5615bf6f43096beb425c895e3f"} Sep 17 18:04:54.069793 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:54.069773 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:04:54.071752 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:54.071732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" event={"ID":"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691","Type":"ContainerStarted","Data":"ac5337d5c1d66bc860c23e3f1aaa3a757a15628029098b40ebe5dd42770e266a"} Sep 17 18:04:54.071928 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:54.071915 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" Sep 17 18:04:54.096186 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:54.096148 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gq22m" podStartSLOduration=2.503187686 podStartE2EDuration="5.096133756s" totalImagesPullingTime="2.59294607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:54.094825323 +0000 UTC m=+119.105012458" watchObservedRunningTime="2026-09-17 18:04:54.096133756 +0000 UTC m=+119.106321267" Sep 17 18:04:54.124636 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:54.124577 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" podStartSLOduration=1.754320731 podStartE2EDuration="5.124559072s" totalImagesPullingTime="3.370238341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:54.123377772 +0000 UTC m=+119.133564908" watchObservedRunningTime="2026-09-17 18:04:54.124559072 +0000 UTC m=+119.134746209" Sep 17 18:04:55.489978 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:55.489948 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:04:56.082328 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.082285 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" event={"ID":"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691","Type":"ContainerStarted","Data":"72f38e18811121ec2523bd9aff474118031f8f8ed96e298cfab9c708c6d79dbf"} Sep 17 18:04:56.082548 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.082370 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" event={"ID":"ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691","Type":"ContainerStarted","Data":"f19c10c29ea96095fb9ac546c550496e61a26c58fad03f6e0a93c9014dfd0c1e"} Sep 17 18:04:56.149384 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.149343 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-68b846668c-m2vs5" podStartSLOduration=2.268694431 podStartE2EDuration="6.149330115s" totalImagesPullingTime="3.880635684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:56.149103886 +0000 UTC m=+121.159291033" watchObservedRunningTime="2026-09-17 18:04:56.149330115 +0000 UTC m=+121.159517251" Sep 17 18:04:56.394260 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.394201 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f7f6bf86-d8tvt"] Sep 17 18:04:56.398690 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.398673 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.414620 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.414600 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f7f6bf86-d8tvt"] Sep 17 18:04:56.448035 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.448003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-oauth-serving-cert\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.448150 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.448040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-trusted-ca-bundle\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.448150 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.448119 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-oauth-config\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.448297 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.448165 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2g26\" (UniqueName: \"kubernetes.io/projected/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-kube-api-access-l2g26\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.448297 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.448207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-config\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.448364 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.448291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-serving-cert\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.448364 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.448331 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-service-ca\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.549383 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.549348 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-service-ca\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.549798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.549398 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-oauth-serving-cert\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.549798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.549424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-trusted-ca-bundle\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.549798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.549482 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-oauth-config\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.549798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.549546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2g26\" (UniqueName: \"kubernetes.io/projected/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-kube-api-access-l2g26\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.549798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.549665 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-config\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.549798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.549709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-serving-cert\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.550199 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.550176 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-service-ca\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.550444 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.550174 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-oauth-serving-cert\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.550444 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.550371 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-config\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.550639 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.550617 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-trusted-ca-bundle\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.552147 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.552129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-serving-cert\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.552228 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.552162 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-oauth-config\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.565340 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.565321 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2g26\" (UniqueName: \"kubernetes.io/projected/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-kube-api-access-l2g26\") pod \"console-7f7f6bf86-d8tvt\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.707863 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.707789 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:04:56.989927 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:56.989833 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f7f6bf86-d8tvt"] Sep 17 18:04:56.992390 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:04:56.992363 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5aa9c65_7b2b_4d4c_8139_cc6fccb6510c.slice/crio-c2e7719e452ab154dde7f15d9c848c36200ec92f8f071925ac677abd835427f8 WatchSource:0}: Error finding container c2e7719e452ab154dde7f15d9c848c36200ec92f8f071925ac677abd835427f8: Status 404 returned error can't find the container with id c2e7719e452ab154dde7f15d9c848c36200ec92f8f071925ac677abd835427f8 Sep 17 18:04:57.089395 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:57.089340 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7f6bf86-d8tvt" event={"ID":"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c","Type":"ContainerStarted","Data":"9373375478fc972e41377d20f42bfc427e709972982d2ab982bc4fc4dd80241c"} Sep 17 18:04:57.089573 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:57.089419 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7f6bf86-d8tvt" event={"ID":"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c","Type":"ContainerStarted","Data":"c2e7719e452ab154dde7f15d9c848c36200ec92f8f071925ac677abd835427f8"} Sep 17 18:04:57.113500 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:57.113452 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f7f6bf86-d8tvt" podStartSLOduration=1.113434689 podStartE2EDuration="1.113434689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:57.112774321 +0000 UTC m=+122.122961457" watchObservedRunningTime="2026-09-17 18:04:57.113434689 +0000 UTC m=+122.123621824" Sep 17 18:04:59.980653 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:04:59.980608 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d97cbd56c-r9w88" Sep 17 18:05:00.082835 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:00.082808 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7b67466fcd-h7hn9" Sep 17 18:05:05.419067 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:05.419032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:05:05.421543 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:05.421495 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d9adda-d73b-4456-83d8-360539054c24-metrics-certs\") pod \"network-metrics-daemon-mt8dc\" (UID: \"18d9adda-d73b-4456-83d8-360539054c24\") " pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:05:05.575297 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:05.575264 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qc24z\"" Sep 17 18:05:05.583960 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:05.583936 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt8dc" Sep 17 18:05:05.867008 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:05.866981 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mt8dc"] Sep 17 18:05:05.870786 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:05:05.870753 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d9adda_d73b_4456_83d8_360539054c24.slice/crio-ea3e1dd855ac305cc587637a82711a7d20c109675baef0f7c11f17fba5c06aa6 WatchSource:0}: Error finding container ea3e1dd855ac305cc587637a82711a7d20c109675baef0f7c11f17fba5c06aa6: Status 404 returned error can't find the container with id ea3e1dd855ac305cc587637a82711a7d20c109675baef0f7c11f17fba5c06aa6 Sep 17 18:05:06.129173 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:06.129099 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mt8dc" event={"ID":"18d9adda-d73b-4456-83d8-360539054c24","Type":"ContainerStarted","Data":"ea3e1dd855ac305cc587637a82711a7d20c109675baef0f7c11f17fba5c06aa6"} Sep 17 18:05:06.709086 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:06.709055 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:05:06.709437 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:06.709108 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:05:06.713733 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:06.713713 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:05:07.138280 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:07.138249 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:05:08.139095 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:08.139055 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mt8dc" event={"ID":"18d9adda-d73b-4456-83d8-360539054c24","Type":"ContainerStarted","Data":"56d07b6c419856a3df24c1eefb16689a29abb5f669aab9cb6188d2fafe8e6cdf"} Sep 17 18:05:08.139433 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:08.139122 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mt8dc" event={"ID":"18d9adda-d73b-4456-83d8-360539054c24","Type":"ContainerStarted","Data":"c44373c6894f598cf1914ffb27a22b02d3a16aaa27ce68d126e133401227554c"} Sep 17 18:05:08.169657 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:08.169617 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mt8dc" podStartSLOduration=131.343488037 podStartE2EDuration="2m13.169601877s" totalImagesPullingTime="1.82611384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:08.169145385 +0000 UTC m=+133.179332521" watchObservedRunningTime="2026-09-17 18:05:08.169601877 +0000 UTC m=+133.179789057" Sep 17 18:05:15.168616 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:15.168585 2564 generic.go:356] "Generic (PLEG): container finished" podID="c3b9adda-9178-46dd-885b-9ffa426d7e03" containerID="99531cf29e5758f6e2a7eb70df157ba0ee5dd370692a99bbe4e4bb2846387d69" exitCode=0 Sep 17 18:05:15.168920 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:15.168659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" event={"ID":"c3b9adda-9178-46dd-885b-9ffa426d7e03","Type":"ContainerDied","Data":"99531cf29e5758f6e2a7eb70df157ba0ee5dd370692a99bbe4e4bb2846387d69"} Sep 17 18:05:15.169012 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:15.168993 2564 scope.go:117] "RemoveContainer" containerID="99531cf29e5758f6e2a7eb70df157ba0ee5dd370692a99bbe4e4bb2846387d69" Sep 17 18:05:16.174103 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:16.174065 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-9xz92" event={"ID":"c3b9adda-9178-46dd-885b-9ffa426d7e03","Type":"ContainerStarted","Data":"944ffadca7925a64a844f48dc7dc8ba4696f26121c7b7214adaeea1409f09c5f"} Sep 17 18:05:17.068917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.068867 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cf6cd8f45-psn52" podUID="f17a0b28-d401-41e9-a051-4911f38043d9" containerName="console" containerID="cri-o://7339d921d41a887658206104d48f24b8af1d14880af84d127355866e3a285a8a" gracePeriod=15 Sep 17 18:05:17.310017 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.309994 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:05:17.410562 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.410511 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f17a0b28-d401-41e9-a051-4911f38043d9-console-oauth-config\") pod \"f17a0b28-d401-41e9-a051-4911f38043d9\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " Sep 17 18:05:17.410562 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.410557 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f17a0b28-d401-41e9-a051-4911f38043d9-console-serving-cert\") pod \"f17a0b28-d401-41e9-a051-4911f38043d9\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " Sep 17 18:05:17.410705 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.410587 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq6m7\" (UniqueName: \"kubernetes.io/projected/f17a0b28-d401-41e9-a051-4911f38043d9-kube-api-access-mq6m7\") pod \"f17a0b28-d401-41e9-a051-4911f38043d9\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " Sep 17 18:05:17.410705 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.410691 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-oauth-serving-cert\") pod \"f17a0b28-d401-41e9-a051-4911f38043d9\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " Sep 17 18:05:17.410794 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.410739 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-trusted-ca-bundle\") pod \"f17a0b28-d401-41e9-a051-4911f38043d9\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " Sep 17 18:05:17.410844 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.410794 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-service-ca\") pod \"f17a0b28-d401-41e9-a051-4911f38043d9\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " Sep 17 18:05:17.410997 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.410949 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-console-config\") pod \"f17a0b28-d401-41e9-a051-4911f38043d9\" (UID: \"f17a0b28-d401-41e9-a051-4911f38043d9\") " Sep 17 18:05:17.411194 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.411132 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-service-ca" (OuterVolumeSpecName: "service-ca") pod "f17a0b28-d401-41e9-a051-4911f38043d9" (UID: "f17a0b28-d401-41e9-a051-4911f38043d9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:05:17.411194 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.411146 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f17a0b28-d401-41e9-a051-4911f38043d9" (UID: "f17a0b28-d401-41e9-a051-4911f38043d9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:05:17.411512 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.411213 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f17a0b28-d401-41e9-a051-4911f38043d9" (UID: "f17a0b28-d401-41e9-a051-4911f38043d9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:05:17.411512 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.411429 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-console-config" (OuterVolumeSpecName: "console-config") pod "f17a0b28-d401-41e9-a051-4911f38043d9" (UID: "f17a0b28-d401-41e9-a051-4911f38043d9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:05:17.411648 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.411575 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-oauth-serving-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:05:17.411648 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.411596 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-trusted-ca-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:05:17.411648 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.411611 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:05:17.411648 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.411625 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f17a0b28-d401-41e9-a051-4911f38043d9-console-config\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:05:17.412945 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.412922 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17a0b28-d401-41e9-a051-4911f38043d9-kube-api-access-mq6m7" (OuterVolumeSpecName: "kube-api-access-mq6m7") pod "f17a0b28-d401-41e9-a051-4911f38043d9" (UID: "f17a0b28-d401-41e9-a051-4911f38043d9"). InnerVolumeSpecName "kube-api-access-mq6m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:05:17.413030 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.412979 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f17a0b28-d401-41e9-a051-4911f38043d9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f17a0b28-d401-41e9-a051-4911f38043d9" (UID: "f17a0b28-d401-41e9-a051-4911f38043d9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:05:17.413030 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.413015 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f17a0b28-d401-41e9-a051-4911f38043d9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f17a0b28-d401-41e9-a051-4911f38043d9" (UID: "f17a0b28-d401-41e9-a051-4911f38043d9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:05:17.512445 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.512418 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f17a0b28-d401-41e9-a051-4911f38043d9-console-oauth-config\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:05:17.512445 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.512441 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f17a0b28-d401-41e9-a051-4911f38043d9-console-serving-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:05:17.512597 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:17.512455 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq6m7\" (UniqueName: \"kubernetes.io/projected/f17a0b28-d401-41e9-a051-4911f38043d9-kube-api-access-mq6m7\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:05:18.183282 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:18.183249 2564 generic.go:356] "Generic (PLEG): container finished" podID="f17a0b28-d401-41e9-a051-4911f38043d9" containerID="7339d921d41a887658206104d48f24b8af1d14880af84d127355866e3a285a8a" exitCode=0 Sep 17 18:05:18.183428 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:18.183307 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf6cd8f45-psn52" Sep 17 18:05:18.183428 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:18.183342 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf6cd8f45-psn52" event={"ID":"f17a0b28-d401-41e9-a051-4911f38043d9","Type":"ContainerDied","Data":"7339d921d41a887658206104d48f24b8af1d14880af84d127355866e3a285a8a"} Sep 17 18:05:18.183428 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:18.183401 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf6cd8f45-psn52" event={"ID":"f17a0b28-d401-41e9-a051-4911f38043d9","Type":"ContainerDied","Data":"2a959c458a48af43131a143963f01dc0dced861ee192aaac51f6e72b37758dfb"} Sep 17 18:05:18.183577 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:18.183429 2564 scope.go:117] "RemoveContainer" containerID="7339d921d41a887658206104d48f24b8af1d14880af84d127355866e3a285a8a" Sep 17 18:05:18.192610 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:18.192581 2564 scope.go:117] "RemoveContainer" containerID="7339d921d41a887658206104d48f24b8af1d14880af84d127355866e3a285a8a" Sep 17 18:05:18.192911 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:05:18.192890 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7339d921d41a887658206104d48f24b8af1d14880af84d127355866e3a285a8a\": container with ID starting with 7339d921d41a887658206104d48f24b8af1d14880af84d127355866e3a285a8a not found: ID does not exist" containerID="7339d921d41a887658206104d48f24b8af1d14880af84d127355866e3a285a8a" Sep 17 18:05:18.192966 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:18.192920 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7339d921d41a887658206104d48f24b8af1d14880af84d127355866e3a285a8a"} err="failed to get container status \"7339d921d41a887658206104d48f24b8af1d14880af84d127355866e3a285a8a\": rpc error: code = NotFound desc = could not find container \"7339d921d41a887658206104d48f24b8af1d14880af84d127355866e3a285a8a\": container with ID starting with 7339d921d41a887658206104d48f24b8af1d14880af84d127355866e3a285a8a not found: ID does not exist" Sep 17 18:05:18.209675 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:18.209648 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cf6cd8f45-psn52"] Sep 17 18:05:18.218258 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:18.218234 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cf6cd8f45-psn52"] Sep 17 18:05:19.449984 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:05:19.449956 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f17a0b28-d401-41e9-a051-4911f38043d9" path="/var/lib/kubelet/pods/f17a0b28-d401-41e9-a051-4911f38043d9/volumes" Sep 17 18:06:31.588404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:31.588366 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f7f6bf86-d8tvt"] Sep 17 18:06:56.609345 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.609306 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f7f6bf86-d8tvt" podUID="d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c" containerName="console" containerID="cri-o://9373375478fc972e41377d20f42bfc427e709972982d2ab982bc4fc4dd80241c" gracePeriod=15 Sep 17 18:06:56.854371 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.854349 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:06:56.906909 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.906845 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-trusted-ca-bundle\") pod \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " Sep 17 18:06:56.906909 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.906892 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-oauth-config\") pod \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " Sep 17 18:06:56.907077 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.906948 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-oauth-serving-cert\") pod \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " Sep 17 18:06:56.907077 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.906987 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-service-ca\") pod \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " Sep 17 18:06:56.907077 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.907023 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-config\") pod \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " Sep 17 18:06:56.907077 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.907055 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2g26\" (UniqueName: \"kubernetes.io/projected/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-kube-api-access-l2g26\") pod \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " Sep 17 18:06:56.907273 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.907100 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-serving-cert\") pod \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\" (UID: \"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c\") " Sep 17 18:06:56.907273 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.907203 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c" (UID: "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:56.907381 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.907352 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-trusted-ca-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:06:56.907434 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.907413 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c" (UID: "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:56.907434 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.907422 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-config" (OuterVolumeSpecName: "console-config") pod "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c" (UID: "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:56.907714 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.907693 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-service-ca" (OuterVolumeSpecName: "service-ca") pod "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c" (UID: "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:56.909152 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.909129 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c" (UID: "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:56.909372 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.909342 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c" (UID: "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:56.909467 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:56.909405 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-kube-api-access-l2g26" (OuterVolumeSpecName: "kube-api-access-l2g26") pod "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c" (UID: "d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c"). InnerVolumeSpecName "kube-api-access-l2g26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:06:57.007660 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.007639 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2g26\" (UniqueName: \"kubernetes.io/projected/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-kube-api-access-l2g26\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:06:57.007660 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.007661 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-serving-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:06:57.007786 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.007670 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-oauth-config\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:06:57.007786 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.007679 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-oauth-serving-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:06:57.007786 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.007687 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:06:57.007786 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.007695 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c-console-config\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:06:57.573310 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.573276 2564 generic.go:356] "Generic (PLEG): container finished" podID="d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c" containerID="9373375478fc972e41377d20f42bfc427e709972982d2ab982bc4fc4dd80241c" exitCode=0 Sep 17 18:06:57.573424 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.573329 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7f6bf86-d8tvt" Sep 17 18:06:57.573424 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.573345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7f6bf86-d8tvt" event={"ID":"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c","Type":"ContainerDied","Data":"9373375478fc972e41377d20f42bfc427e709972982d2ab982bc4fc4dd80241c"} Sep 17 18:06:57.573424 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.573392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7f6bf86-d8tvt" event={"ID":"d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c","Type":"ContainerDied","Data":"c2e7719e452ab154dde7f15d9c848c36200ec92f8f071925ac677abd835427f8"} Sep 17 18:06:57.573424 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.573419 2564 scope.go:117] "RemoveContainer" containerID="9373375478fc972e41377d20f42bfc427e709972982d2ab982bc4fc4dd80241c" Sep 17 18:06:57.581605 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.581588 2564 scope.go:117] "RemoveContainer" containerID="9373375478fc972e41377d20f42bfc427e709972982d2ab982bc4fc4dd80241c" Sep 17 18:06:57.581889 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:06:57.581865 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9373375478fc972e41377d20f42bfc427e709972982d2ab982bc4fc4dd80241c\": container with ID starting with 9373375478fc972e41377d20f42bfc427e709972982d2ab982bc4fc4dd80241c not found: ID does not exist" containerID="9373375478fc972e41377d20f42bfc427e709972982d2ab982bc4fc4dd80241c" Sep 17 18:06:57.581960 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.581898 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9373375478fc972e41377d20f42bfc427e709972982d2ab982bc4fc4dd80241c"} err="failed to get container status \"9373375478fc972e41377d20f42bfc427e709972982d2ab982bc4fc4dd80241c\": rpc error: code = NotFound desc = could not find container \"9373375478fc972e41377d20f42bfc427e709972982d2ab982bc4fc4dd80241c\": container with ID starting with 9373375478fc972e41377d20f42bfc427e709972982d2ab982bc4fc4dd80241c not found: ID does not exist" Sep 17 18:06:57.594824 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.594803 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f7f6bf86-d8tvt"] Sep 17 18:06:57.600327 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:57.600306 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f7f6bf86-d8tvt"] Sep 17 18:06:59.450421 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:06:59.450376 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c" path="/var/lib/kubelet/pods/d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c/volumes" Sep 17 18:07:01.138438 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.138386 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx"] Sep 17 18:07:01.139751 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.139712 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f17a0b28-d401-41e9-a051-4911f38043d9" containerName="console" Sep 17 18:07:01.139945 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.139932 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f17a0b28-d401-41e9-a051-4911f38043d9" containerName="console" Sep 17 18:07:01.140031 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.140021 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c" containerName="console" Sep 17 18:07:01.140095 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.140087 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c" containerName="console" Sep 17 18:07:01.140357 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.140344 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f17a0b28-d401-41e9-a051-4911f38043d9" containerName="console" Sep 17 18:07:01.140458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.140448 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5aa9c65-7b2b-4d4c-8139-cc6fccb6510c" containerName="console" Sep 17 18:07:01.145990 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.145957 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" Sep 17 18:07:01.154317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.154291 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 18:07:01.154317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.154311 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 18:07:01.155194 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.155172 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vwsz8\"" Sep 17 18:07:01.162341 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.162314 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx"] Sep 17 18:07:01.235901 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.235875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc9590c2-1440-4110-91cc-3a0b03e6acbc-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx\" (UID: \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" Sep 17 18:07:01.235988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.235908 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc9590c2-1440-4110-91cc-3a0b03e6acbc-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx\" (UID: \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" Sep 17 18:07:01.235988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.235946 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8hx9\" (UniqueName: \"kubernetes.io/projected/fc9590c2-1440-4110-91cc-3a0b03e6acbc-kube-api-access-b8hx9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx\" (UID: \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" Sep 17 18:07:01.336491 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.336462 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc9590c2-1440-4110-91cc-3a0b03e6acbc-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx\" (UID: \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" Sep 17 18:07:01.336616 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.336499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc9590c2-1440-4110-91cc-3a0b03e6acbc-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx\" (UID: \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" Sep 17 18:07:01.336616 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.336556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8hx9\" (UniqueName: \"kubernetes.io/projected/fc9590c2-1440-4110-91cc-3a0b03e6acbc-kube-api-access-b8hx9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx\" (UID: \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" Sep 17 18:07:01.336971 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.336944 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc9590c2-1440-4110-91cc-3a0b03e6acbc-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx\" (UID: \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" Sep 17 18:07:01.337007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.336954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc9590c2-1440-4110-91cc-3a0b03e6acbc-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx\" (UID: \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" Sep 17 18:07:01.345871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.345839 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8hx9\" (UniqueName: \"kubernetes.io/projected/fc9590c2-1440-4110-91cc-3a0b03e6acbc-kube-api-access-b8hx9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx\" (UID: \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" Sep 17 18:07:01.454965 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.454912 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" Sep 17 18:07:01.732580 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:01.732548 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx"] Sep 17 18:07:01.734955 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:07:01.734927 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9590c2_1440_4110_91cc_3a0b03e6acbc.slice/crio-2b8ecfeaa8747af2d5b2d52d7ee270b8ff43dabb9e345b7665eae404f2ff5229 WatchSource:0}: Error finding container 2b8ecfeaa8747af2d5b2d52d7ee270b8ff43dabb9e345b7665eae404f2ff5229: Status 404 returned error can't find the container with id 2b8ecfeaa8747af2d5b2d52d7ee270b8ff43dabb9e345b7665eae404f2ff5229 Sep 17 18:07:02.596831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:02.596791 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" event={"ID":"fc9590c2-1440-4110-91cc-3a0b03e6acbc","Type":"ContainerStarted","Data":"2b8ecfeaa8747af2d5b2d52d7ee270b8ff43dabb9e345b7665eae404f2ff5229"} Sep 17 18:07:09.628721 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:09.628682 2564 generic.go:356] "Generic (PLEG): container finished" podID="fc9590c2-1440-4110-91cc-3a0b03e6acbc" containerID="49ff3c525621b5bc62fa82dbd310c68f4a4fb515780c67222b5b9b1ff156f8e4" exitCode=0 Sep 17 18:07:09.629089 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:09.628768 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" event={"ID":"fc9590c2-1440-4110-91cc-3a0b03e6acbc","Type":"ContainerDied","Data":"49ff3c525621b5bc62fa82dbd310c68f4a4fb515780c67222b5b9b1ff156f8e4"} Sep 17 18:07:12.648674 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:12.648642 2564 generic.go:356] "Generic (PLEG): container finished" podID="fc9590c2-1440-4110-91cc-3a0b03e6acbc" containerID="75f7700e6e916cb52c09b27f74cce44696f93add1bd4adcd3e4681acab4d5e41" exitCode=0 Sep 17 18:07:12.649022 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:12.648702 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" event={"ID":"fc9590c2-1440-4110-91cc-3a0b03e6acbc","Type":"ContainerDied","Data":"75f7700e6e916cb52c09b27f74cce44696f93add1bd4adcd3e4681acab4d5e41"} Sep 17 18:07:20.684387 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:20.684352 2564 generic.go:356] "Generic (PLEG): container finished" podID="fc9590c2-1440-4110-91cc-3a0b03e6acbc" containerID="3f4a7469d51c6ced5a30ad232e7675dc963502647c4c2876b551c3c46f1ca9de" exitCode=0 Sep 17 18:07:20.684775 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:20.684421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" event={"ID":"fc9590c2-1440-4110-91cc-3a0b03e6acbc","Type":"ContainerDied","Data":"3f4a7469d51c6ced5a30ad232e7675dc963502647c4c2876b551c3c46f1ca9de"} Sep 17 18:07:21.808538 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:21.808504 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" Sep 17 18:07:21.892236 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:21.892211 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc9590c2-1440-4110-91cc-3a0b03e6acbc-bundle\") pod \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\" (UID: \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\") " Sep 17 18:07:21.892342 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:21.892296 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8hx9\" (UniqueName: \"kubernetes.io/projected/fc9590c2-1440-4110-91cc-3a0b03e6acbc-kube-api-access-b8hx9\") pod \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\" (UID: \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\") " Sep 17 18:07:21.892342 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:21.892326 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc9590c2-1440-4110-91cc-3a0b03e6acbc-util\") pod \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\" (UID: \"fc9590c2-1440-4110-91cc-3a0b03e6acbc\") " Sep 17 18:07:21.892902 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:21.892876 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9590c2-1440-4110-91cc-3a0b03e6acbc-bundle" (OuterVolumeSpecName: "bundle") pod "fc9590c2-1440-4110-91cc-3a0b03e6acbc" (UID: "fc9590c2-1440-4110-91cc-3a0b03e6acbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:07:21.894464 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:21.894434 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9590c2-1440-4110-91cc-3a0b03e6acbc-kube-api-access-b8hx9" (OuterVolumeSpecName: "kube-api-access-b8hx9") pod "fc9590c2-1440-4110-91cc-3a0b03e6acbc" (UID: "fc9590c2-1440-4110-91cc-3a0b03e6acbc"). InnerVolumeSpecName "kube-api-access-b8hx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:07:21.898192 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:21.898168 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9590c2-1440-4110-91cc-3a0b03e6acbc-util" (OuterVolumeSpecName: "util") pod "fc9590c2-1440-4110-91cc-3a0b03e6acbc" (UID: "fc9590c2-1440-4110-91cc-3a0b03e6acbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:07:21.993178 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:21.993127 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc9590c2-1440-4110-91cc-3a0b03e6acbc-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:07:21.993178 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:21.993145 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc9590c2-1440-4110-91cc-3a0b03e6acbc-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:07:21.993178 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:21.993155 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8hx9\" (UniqueName: \"kubernetes.io/projected/fc9590c2-1440-4110-91cc-3a0b03e6acbc-kube-api-access-b8hx9\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:07:22.694972 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:22.694941 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" Sep 17 18:07:22.694972 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:22.694956 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407z6qx" event={"ID":"fc9590c2-1440-4110-91cc-3a0b03e6acbc","Type":"ContainerDied","Data":"2b8ecfeaa8747af2d5b2d52d7ee270b8ff43dabb9e345b7665eae404f2ff5229"} Sep 17 18:07:22.695156 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:22.694994 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b8ecfeaa8747af2d5b2d52d7ee270b8ff43dabb9e345b7665eae404f2ff5229" Sep 17 18:07:29.352970 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.352931 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8"] Sep 17 18:07:29.353342 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.353219 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9590c2-1440-4110-91cc-3a0b03e6acbc" containerName="extract" Sep 17 18:07:29.353342 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.353231 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9590c2-1440-4110-91cc-3a0b03e6acbc" containerName="extract" Sep 17 18:07:29.353342 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.353242 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9590c2-1440-4110-91cc-3a0b03e6acbc" containerName="util" Sep 17 18:07:29.353342 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.353248 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9590c2-1440-4110-91cc-3a0b03e6acbc" containerName="util" Sep 17 18:07:29.353342 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.353268 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9590c2-1440-4110-91cc-3a0b03e6acbc" containerName="pull" Sep 17 18:07:29.353342 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.353274 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9590c2-1440-4110-91cc-3a0b03e6acbc" containerName="pull" Sep 17 18:07:29.353342 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.353324 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc9590c2-1440-4110-91cc-3a0b03e6acbc" containerName="extract" Sep 17 18:07:29.356690 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.356672 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8" Sep 17 18:07:29.360822 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.360799 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 17 18:07:29.360822 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.360818 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-f6tsw\"" Sep 17 18:07:29.361191 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.361175 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:07:29.376364 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.376340 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8"] Sep 17 18:07:29.442297 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.442273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e9e3dcf-31a4-4372-8039-abc64089efc2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t95m8\" (UID: \"4e9e3dcf-31a4-4372-8039-abc64089efc2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8" Sep 17 18:07:29.442394 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.442303 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q27hh\" (UniqueName: \"kubernetes.io/projected/4e9e3dcf-31a4-4372-8039-abc64089efc2-kube-api-access-q27hh\") pod \"cert-manager-operator-controller-manager-55c444bf86-t95m8\" (UID: \"4e9e3dcf-31a4-4372-8039-abc64089efc2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8" Sep 17 18:07:29.542654 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.542618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e9e3dcf-31a4-4372-8039-abc64089efc2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t95m8\" (UID: \"4e9e3dcf-31a4-4372-8039-abc64089efc2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8" Sep 17 18:07:29.542654 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.542661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q27hh\" (UniqueName: \"kubernetes.io/projected/4e9e3dcf-31a4-4372-8039-abc64089efc2-kube-api-access-q27hh\") pod \"cert-manager-operator-controller-manager-55c444bf86-t95m8\" (UID: \"4e9e3dcf-31a4-4372-8039-abc64089efc2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8" Sep 17 18:07:29.543004 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.542987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e9e3dcf-31a4-4372-8039-abc64089efc2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t95m8\" (UID: \"4e9e3dcf-31a4-4372-8039-abc64089efc2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8" Sep 17 18:07:29.551581 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.551556 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q27hh\" (UniqueName: \"kubernetes.io/projected/4e9e3dcf-31a4-4372-8039-abc64089efc2-kube-api-access-q27hh\") pod \"cert-manager-operator-controller-manager-55c444bf86-t95m8\" (UID: \"4e9e3dcf-31a4-4372-8039-abc64089efc2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8" Sep 17 18:07:29.665938 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.665886 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8" Sep 17 18:07:29.951653 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:29.951583 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8"] Sep 17 18:07:29.955423 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:07:29.955397 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9e3dcf_31a4_4372_8039_abc64089efc2.slice/crio-d8533cc7326bed1655ab49227964a3b7ca60829fe1941c1718762e7798e8a5cf WatchSource:0}: Error finding container d8533cc7326bed1655ab49227964a3b7ca60829fe1941c1718762e7798e8a5cf: Status 404 returned error can't find the container with id d8533cc7326bed1655ab49227964a3b7ca60829fe1941c1718762e7798e8a5cf Sep 17 18:07:30.728703 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:30.728665 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8" event={"ID":"4e9e3dcf-31a4-4372-8039-abc64089efc2","Type":"ContainerStarted","Data":"d8533cc7326bed1655ab49227964a3b7ca60829fe1941c1718762e7798e8a5cf"} Sep 17 18:07:31.734459 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:31.734391 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8" event={"ID":"4e9e3dcf-31a4-4372-8039-abc64089efc2","Type":"ContainerStarted","Data":"064b125c8d8cb881f479309a7ee30cade19c4fa06f51fb1a0a678a3a4fe60c8b"} Sep 17 18:07:31.770228 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:31.770182 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t95m8" podStartSLOduration=1.253273723 podStartE2EDuration="2.770168099s" totalImagesPullingTime="1.516894376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:31.769893848 +0000 UTC m=+276.780080987" watchObservedRunningTime="2026-09-17 18:07:31.770168099 +0000 UTC m=+276.780355234" Sep 17 18:07:33.202754 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.202722 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk"] Sep 17 18:07:33.205263 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.205244 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" Sep 17 18:07:33.208028 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.208002 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 18:07:33.208904 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.208880 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vwsz8\"" Sep 17 18:07:33.209019 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.208880 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 18:07:33.220364 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.220342 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk"] Sep 17 18:07:33.273562 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.273505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/025d4620-0b88-42dd-9e22-026194318076-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk\" (UID: \"025d4620-0b88-42dd-9e22-026194318076\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" Sep 17 18:07:33.273673 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.273577 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/025d4620-0b88-42dd-9e22-026194318076-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk\" (UID: \"025d4620-0b88-42dd-9e22-026194318076\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" Sep 17 18:07:33.273673 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.273617 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8tcc\" (UniqueName: \"kubernetes.io/projected/025d4620-0b88-42dd-9e22-026194318076-kube-api-access-g8tcc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk\" (UID: \"025d4620-0b88-42dd-9e22-026194318076\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" Sep 17 18:07:33.374226 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.374198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/025d4620-0b88-42dd-9e22-026194318076-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk\" (UID: \"025d4620-0b88-42dd-9e22-026194318076\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" Sep 17 18:07:33.374321 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.374238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/025d4620-0b88-42dd-9e22-026194318076-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk\" (UID: \"025d4620-0b88-42dd-9e22-026194318076\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" Sep 17 18:07:33.374321 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.374273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8tcc\" (UniqueName: \"kubernetes.io/projected/025d4620-0b88-42dd-9e22-026194318076-kube-api-access-g8tcc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk\" (UID: \"025d4620-0b88-42dd-9e22-026194318076\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" Sep 17 18:07:33.374666 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.374641 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/025d4620-0b88-42dd-9e22-026194318076-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk\" (UID: \"025d4620-0b88-42dd-9e22-026194318076\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" Sep 17 18:07:33.374724 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.374671 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/025d4620-0b88-42dd-9e22-026194318076-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk\" (UID: \"025d4620-0b88-42dd-9e22-026194318076\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" Sep 17 18:07:33.382931 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.382906 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8tcc\" (UniqueName: \"kubernetes.io/projected/025d4620-0b88-42dd-9e22-026194318076-kube-api-access-g8tcc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk\" (UID: \"025d4620-0b88-42dd-9e22-026194318076\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" Sep 17 18:07:33.519222 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.519164 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" Sep 17 18:07:33.797119 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:33.797096 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk"] Sep 17 18:07:33.798777 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:07:33.798750 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025d4620_0b88_42dd_9e22_026194318076.slice/crio-b02986d410c0506e9afaa93355dc4c12e9890efe8eae786e907de700bbf9c104 WatchSource:0}: Error finding container b02986d410c0506e9afaa93355dc4c12e9890efe8eae786e907de700bbf9c104: Status 404 returned error can't find the container with id b02986d410c0506e9afaa93355dc4c12e9890efe8eae786e907de700bbf9c104 Sep 17 18:07:34.751248 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:34.751163 2564 generic.go:356] "Generic (PLEG): container finished" podID="025d4620-0b88-42dd-9e22-026194318076" containerID="c77b789b0ead744e0e1dd643c4822605126d6204814cce609d9536c60877c3d0" exitCode=0 Sep 17 18:07:34.751614 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:34.751265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" event={"ID":"025d4620-0b88-42dd-9e22-026194318076","Type":"ContainerDied","Data":"c77b789b0ead744e0e1dd643c4822605126d6204814cce609d9536c60877c3d0"} Sep 17 18:07:34.751614 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:34.751345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" event={"ID":"025d4620-0b88-42dd-9e22-026194318076","Type":"ContainerStarted","Data":"b02986d410c0506e9afaa93355dc4c12e9890efe8eae786e907de700bbf9c104"} Sep 17 18:07:35.184477 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.184446 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk"] Sep 17 18:07:35.186787 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.186772 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk" Sep 17 18:07:35.189463 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.189442 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5hphb\"" Sep 17 18:07:35.189463 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.189444 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 17 18:07:35.189896 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.189879 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 17 18:07:35.197928 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.197905 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk"] Sep 17 18:07:35.287216 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.287194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3224e502-c5a3-4621-bcf7-784b065f5a9a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6w9kk\" (UID: \"3224e502-c5a3-4621-bcf7-784b065f5a9a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk" Sep 17 18:07:35.287322 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.287248 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzkhg\" (UniqueName: \"kubernetes.io/projected/3224e502-c5a3-4621-bcf7-784b065f5a9a-kube-api-access-hzkhg\") pod \"cert-manager-cainjector-559cdd8c87-6w9kk\" (UID: \"3224e502-c5a3-4621-bcf7-784b065f5a9a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk" Sep 17 18:07:35.387929 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.387906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3224e502-c5a3-4621-bcf7-784b065f5a9a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6w9kk\" (UID: \"3224e502-c5a3-4621-bcf7-784b065f5a9a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk" Sep 17 18:07:35.388019 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.387949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzkhg\" (UniqueName: \"kubernetes.io/projected/3224e502-c5a3-4621-bcf7-784b065f5a9a-kube-api-access-hzkhg\") pod \"cert-manager-cainjector-559cdd8c87-6w9kk\" (UID: \"3224e502-c5a3-4621-bcf7-784b065f5a9a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk" Sep 17 18:07:35.396740 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.396717 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3224e502-c5a3-4621-bcf7-784b065f5a9a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6w9kk\" (UID: \"3224e502-c5a3-4621-bcf7-784b065f5a9a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk" Sep 17 18:07:35.396837 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.396822 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzkhg\" (UniqueName: \"kubernetes.io/projected/3224e502-c5a3-4621-bcf7-784b065f5a9a-kube-api-access-hzkhg\") pod \"cert-manager-cainjector-559cdd8c87-6w9kk\" (UID: \"3224e502-c5a3-4621-bcf7-784b065f5a9a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk" Sep 17 18:07:35.496033 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.495981 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk" Sep 17 18:07:35.776821 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:35.776795 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk"] Sep 17 18:07:35.778727 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:07:35.778690 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3224e502_c5a3_4621_bcf7_784b065f5a9a.slice/crio-7b7fbbb050ff84488748f8618e363025b906f8542293b98ecc6841bdffcb5f8d WatchSource:0}: Error finding container 7b7fbbb050ff84488748f8618e363025b906f8542293b98ecc6841bdffcb5f8d: Status 404 returned error can't find the container with id 7b7fbbb050ff84488748f8618e363025b906f8542293b98ecc6841bdffcb5f8d Sep 17 18:07:36.762145 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:36.762104 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk" event={"ID":"3224e502-c5a3-4621-bcf7-784b065f5a9a","Type":"ContainerStarted","Data":"7b7fbbb050ff84488748f8618e363025b906f8542293b98ecc6841bdffcb5f8d"} Sep 17 18:07:37.769260 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:37.769227 2564 generic.go:356] "Generic (PLEG): container finished" podID="025d4620-0b88-42dd-9e22-026194318076" containerID="d30eaddf5bd11b75e56036267e98334e4bde9b41c7141b2c90921a4fbbbd5963" exitCode=0 Sep 17 18:07:37.769701 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:37.769288 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" event={"ID":"025d4620-0b88-42dd-9e22-026194318076","Type":"ContainerDied","Data":"d30eaddf5bd11b75e56036267e98334e4bde9b41c7141b2c90921a4fbbbd5963"} Sep 17 18:07:38.776610 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:38.776572 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" event={"ID":"025d4620-0b88-42dd-9e22-026194318076","Type":"ContainerStarted","Data":"57d91b9097cb4fa39a4850df9d1ea21b98fff3d58795e02473890ae05c50d5c0"} Sep 17 18:07:39.783450 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:39.783418 2564 generic.go:356] "Generic (PLEG): container finished" podID="025d4620-0b88-42dd-9e22-026194318076" containerID="57d91b9097cb4fa39a4850df9d1ea21b98fff3d58795e02473890ae05c50d5c0" exitCode=0 Sep 17 18:07:39.783870 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:39.783509 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" event={"ID":"025d4620-0b88-42dd-9e22-026194318076","Type":"ContainerDied","Data":"57d91b9097cb4fa39a4850df9d1ea21b98fff3d58795e02473890ae05c50d5c0"} Sep 17 18:07:39.785010 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:39.784990 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk" event={"ID":"3224e502-c5a3-4621-bcf7-784b065f5a9a","Type":"ContainerStarted","Data":"cc1da54465536004ed9a2622bcab87e00df446f266d91342fad900fc2d4a4600"} Sep 17 18:07:39.832033 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:39.831991 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6w9kk" podStartSLOduration=1.780952245 podStartE2EDuration="4.831979648s" totalImagesPullingTime="3.051027403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:39.830167254 +0000 UTC m=+284.840354389" watchObservedRunningTime="2026-09-17 18:07:39.831979648 +0000 UTC m=+284.842166781" Sep 17 18:07:40.910476 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:40.910454 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" Sep 17 18:07:41.036226 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:41.036195 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/025d4620-0b88-42dd-9e22-026194318076-bundle\") pod \"025d4620-0b88-42dd-9e22-026194318076\" (UID: \"025d4620-0b88-42dd-9e22-026194318076\") " Sep 17 18:07:41.036226 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:41.036228 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/025d4620-0b88-42dd-9e22-026194318076-util\") pod \"025d4620-0b88-42dd-9e22-026194318076\" (UID: \"025d4620-0b88-42dd-9e22-026194318076\") " Sep 17 18:07:41.036396 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:41.036275 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8tcc\" (UniqueName: \"kubernetes.io/projected/025d4620-0b88-42dd-9e22-026194318076-kube-api-access-g8tcc\") pod \"025d4620-0b88-42dd-9e22-026194318076\" (UID: \"025d4620-0b88-42dd-9e22-026194318076\") " Sep 17 18:07:41.036599 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:41.036568 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025d4620-0b88-42dd-9e22-026194318076-bundle" (OuterVolumeSpecName: "bundle") pod "025d4620-0b88-42dd-9e22-026194318076" (UID: "025d4620-0b88-42dd-9e22-026194318076"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:07:41.038562 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:41.038540 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025d4620-0b88-42dd-9e22-026194318076-kube-api-access-g8tcc" (OuterVolumeSpecName: "kube-api-access-g8tcc") pod "025d4620-0b88-42dd-9e22-026194318076" (UID: "025d4620-0b88-42dd-9e22-026194318076"). InnerVolumeSpecName "kube-api-access-g8tcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:07:41.041195 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:41.041151 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025d4620-0b88-42dd-9e22-026194318076-util" (OuterVolumeSpecName: "util") pod "025d4620-0b88-42dd-9e22-026194318076" (UID: "025d4620-0b88-42dd-9e22-026194318076"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:07:41.137765 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:41.137737 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/025d4620-0b88-42dd-9e22-026194318076-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:07:41.137765 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:41.137761 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/025d4620-0b88-42dd-9e22-026194318076-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:07:41.137894 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:41.137774 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8tcc\" (UniqueName: \"kubernetes.io/projected/025d4620-0b88-42dd-9e22-026194318076-kube-api-access-g8tcc\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:07:41.796626 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:41.796560 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" event={"ID":"025d4620-0b88-42dd-9e22-026194318076","Type":"ContainerDied","Data":"b02986d410c0506e9afaa93355dc4c12e9890efe8eae786e907de700bbf9c104"} Sep 17 18:07:41.796626 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:41.796580 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l2lk" Sep 17 18:07:41.796779 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:41.796637 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02986d410c0506e9afaa93355dc4c12e9890efe8eae786e907de700bbf9c104" Sep 17 18:07:52.974612 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:52.974579 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj"] Sep 17 18:07:52.975028 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:52.974853 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025d4620-0b88-42dd-9e22-026194318076" containerName="extract" Sep 17 18:07:52.975028 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:52.974863 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="025d4620-0b88-42dd-9e22-026194318076" containerName="extract" Sep 17 18:07:52.975028 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:52.974874 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025d4620-0b88-42dd-9e22-026194318076" containerName="pull" Sep 17 18:07:52.975028 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:52.974880 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="025d4620-0b88-42dd-9e22-026194318076" containerName="pull" Sep 17 18:07:52.975028 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:52.974888 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025d4620-0b88-42dd-9e22-026194318076" containerName="util" Sep 17 18:07:52.975028 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:52.974893 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="025d4620-0b88-42dd-9e22-026194318076" containerName="util" Sep 17 18:07:52.975028 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:52.974950 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="025d4620-0b88-42dd-9e22-026194318076" containerName="extract" Sep 17 18:07:53.012613 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.012590 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj"] Sep 17 18:07:53.012734 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.012679 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" Sep 17 18:07:53.015411 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.015386 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 18:07:53.015642 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.015627 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vwsz8\"" Sep 17 18:07:53.016348 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.016334 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 18:07:53.121236 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.121213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj\" (UID: \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" Sep 17 18:07:53.121335 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.121265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d275r\" (UniqueName: \"kubernetes.io/projected/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-kube-api-access-d275r\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj\" (UID: \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" Sep 17 18:07:53.121335 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.121305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj\" (UID: \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" Sep 17 18:07:53.221815 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.221794 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d275r\" (UniqueName: \"kubernetes.io/projected/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-kube-api-access-d275r\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj\" (UID: \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" Sep 17 18:07:53.221921 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.221825 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj\" (UID: \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" Sep 17 18:07:53.221921 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.221859 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj\" (UID: \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" Sep 17 18:07:53.222158 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.222141 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj\" (UID: \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" Sep 17 18:07:53.222194 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.222174 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj\" (UID: \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" Sep 17 18:07:53.244223 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.244167 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d275r\" (UniqueName: \"kubernetes.io/projected/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-kube-api-access-d275r\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj\" (UID: \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" Sep 17 18:07:53.321084 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.321068 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" Sep 17 18:07:53.596562 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.596511 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj"] Sep 17 18:07:53.601412 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:07:53.601391 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9bd0f1_ec52_422b_bb88_834e7d7571d6.slice/crio-d31b9b14d8d22fc57f580f80649cb6fbe607df354c5d57b5997aadbfb50f77bd WatchSource:0}: Error finding container d31b9b14d8d22fc57f580f80649cb6fbe607df354c5d57b5997aadbfb50f77bd: Status 404 returned error can't find the container with id d31b9b14d8d22fc57f580f80649cb6fbe607df354c5d57b5997aadbfb50f77bd Sep 17 18:07:53.849281 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.849216 2564 generic.go:356] "Generic (PLEG): container finished" podID="4b9bd0f1-ec52-422b-bb88-834e7d7571d6" containerID="c0d721959f3a1125afc9e8a2a6eda38a4346ce688bbe298c5e9ae55704b0154d" exitCode=0 Sep 17 18:07:53.849388 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.849306 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" event={"ID":"4b9bd0f1-ec52-422b-bb88-834e7d7571d6","Type":"ContainerDied","Data":"c0d721959f3a1125afc9e8a2a6eda38a4346ce688bbe298c5e9ae55704b0154d"} Sep 17 18:07:53.849388 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:53.849363 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" event={"ID":"4b9bd0f1-ec52-422b-bb88-834e7d7571d6","Type":"ContainerStarted","Data":"d31b9b14d8d22fc57f580f80649cb6fbe607df354c5d57b5997aadbfb50f77bd"} Sep 17 18:07:54.854757 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:54.854725 2564 generic.go:356] "Generic (PLEG): container finished" podID="4b9bd0f1-ec52-422b-bb88-834e7d7571d6" containerID="d2c1a8c67311a4312218b25f937190e23b0981c878e0d0224c6c7e127ab24ec5" exitCode=0 Sep 17 18:07:54.855129 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:54.854770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" event={"ID":"4b9bd0f1-ec52-422b-bb88-834e7d7571d6","Type":"ContainerDied","Data":"d2c1a8c67311a4312218b25f937190e23b0981c878e0d0224c6c7e127ab24ec5"} Sep 17 18:07:55.367399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:55.367373 2564 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 18:07:55.860437 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:55.860406 2564 generic.go:356] "Generic (PLEG): container finished" podID="4b9bd0f1-ec52-422b-bb88-834e7d7571d6" containerID="d48906ca82151bead9ed0e80ddb3bd1cff46223a36ea5c06e125514bb7e779b9" exitCode=0 Sep 17 18:07:55.861813 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:55.860443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" event={"ID":"4b9bd0f1-ec52-422b-bb88-834e7d7571d6","Type":"ContainerDied","Data":"d48906ca82151bead9ed0e80ddb3bd1cff46223a36ea5c06e125514bb7e779b9"} Sep 17 18:07:56.993997 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:56.993973 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" Sep 17 18:07:57.151824 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:57.151758 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d275r\" (UniqueName: \"kubernetes.io/projected/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-kube-api-access-d275r\") pod \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\" (UID: \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\") " Sep 17 18:07:57.151951 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:57.151850 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-util\") pod \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\" (UID: \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\") " Sep 17 18:07:57.151951 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:57.151879 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-bundle\") pod \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\" (UID: \"4b9bd0f1-ec52-422b-bb88-834e7d7571d6\") " Sep 17 18:07:57.152622 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:57.152569 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-bundle" (OuterVolumeSpecName: "bundle") pod "4b9bd0f1-ec52-422b-bb88-834e7d7571d6" (UID: "4b9bd0f1-ec52-422b-bb88-834e7d7571d6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:07:57.154022 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:57.153989 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-kube-api-access-d275r" (OuterVolumeSpecName: "kube-api-access-d275r") pod "4b9bd0f1-ec52-422b-bb88-834e7d7571d6" (UID: "4b9bd0f1-ec52-422b-bb88-834e7d7571d6"). InnerVolumeSpecName "kube-api-access-d275r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:07:57.158184 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:57.158126 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-util" (OuterVolumeSpecName: "util") pod "4b9bd0f1-ec52-422b-bb88-834e7d7571d6" (UID: "4b9bd0f1-ec52-422b-bb88-834e7d7571d6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:07:57.252335 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:57.252309 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:07:57.252335 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:57.252331 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:07:57.252465 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:57.252342 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d275r\" (UniqueName: \"kubernetes.io/projected/4b9bd0f1-ec52-422b-bb88-834e7d7571d6-kube-api-access-d275r\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:07:57.876426 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:57.876402 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" Sep 17 18:07:57.876580 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:57.876428 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrpktj" event={"ID":"4b9bd0f1-ec52-422b-bb88-834e7d7571d6","Type":"ContainerDied","Data":"d31b9b14d8d22fc57f580f80649cb6fbe607df354c5d57b5997aadbfb50f77bd"} Sep 17 18:07:57.876580 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:07:57.876472 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d31b9b14d8d22fc57f580f80649cb6fbe607df354c5d57b5997aadbfb50f77bd" Sep 17 18:08:07.210912 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.210869 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm"] Sep 17 18:08:07.211385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.211315 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b9bd0f1-ec52-422b-bb88-834e7d7571d6" containerName="util" Sep 17 18:08:07.211385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.211333 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b9bd0f1-ec52-422b-bb88-834e7d7571d6" containerName="util" Sep 17 18:08:07.211385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.211350 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b9bd0f1-ec52-422b-bb88-834e7d7571d6" containerName="extract" Sep 17 18:08:07.211385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.211357 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b9bd0f1-ec52-422b-bb88-834e7d7571d6" containerName="extract" Sep 17 18:08:07.211385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.211370 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b9bd0f1-ec52-422b-bb88-834e7d7571d6" containerName="pull" Sep 17 18:08:07.211385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.211378 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b9bd0f1-ec52-422b-bb88-834e7d7571d6" containerName="pull" Sep 17 18:08:07.211737 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.211478 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b9bd0f1-ec52-422b-bb88-834e7d7571d6" containerName="extract" Sep 17 18:08:07.213951 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.213929 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.218798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.218780 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 17 18:08:07.218798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.218793 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 17 18:08:07.219153 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.219134 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8977h\"" Sep 17 18:08:07.219419 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.219403 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 17 18:08:07.219508 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.219478 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 17 18:08:07.233584 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.233541 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm"] Sep 17 18:08:07.312761 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.312736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7093f3f1-2b75-40b1-8148-f70d97f35fb2-manifests\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.312888 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.312776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vpk\" (UniqueName: \"kubernetes.io/projected/7093f3f1-2b75-40b1-8148-f70d97f35fb2-kube-api-access-55vpk\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.312888 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.312864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7093f3f1-2b75-40b1-8148-f70d97f35fb2-webhook-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.312996 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.312905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7093f3f1-2b75-40b1-8148-f70d97f35fb2-tmp\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.312996 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.312943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7093f3f1-2b75-40b1-8148-f70d97f35fb2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.407874 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.407850 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt"] Sep 17 18:08:07.410391 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.410377 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" Sep 17 18:08:07.413247 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.413226 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 18:08:07.413352 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.413264 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45pt9\" (UniqueName: \"kubernetes.io/projected/e8d60341-160c-4382-8e7d-a5290f1a49b6-kube-api-access-45pt9\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt\" (UID: \"e8d60341-160c-4382-8e7d-a5290f1a49b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" Sep 17 18:08:07.413352 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.413289 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 18:08:07.413352 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.413319 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vwsz8\"" Sep 17 18:08:07.413577 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.413342 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7093f3f1-2b75-40b1-8148-f70d97f35fb2-tmp\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.413577 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.413423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7093f3f1-2b75-40b1-8148-f70d97f35fb2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.413577 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.413472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7093f3f1-2b75-40b1-8148-f70d97f35fb2-manifests\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.413577 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.413500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55vpk\" (UniqueName: \"kubernetes.io/projected/7093f3f1-2b75-40b1-8148-f70d97f35fb2-kube-api-access-55vpk\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.413775 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.413578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7093f3f1-2b75-40b1-8148-f70d97f35fb2-webhook-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.413775 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.413607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8d60341-160c-4382-8e7d-a5290f1a49b6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt\" (UID: \"e8d60341-160c-4382-8e7d-a5290f1a49b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" Sep 17 18:08:07.413775 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.413633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8d60341-160c-4382-8e7d-a5290f1a49b6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt\" (UID: \"e8d60341-160c-4382-8e7d-a5290f1a49b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" Sep 17 18:08:07.413913 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.413889 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7093f3f1-2b75-40b1-8148-f70d97f35fb2-manifests\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.415981 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.415958 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7093f3f1-2b75-40b1-8148-f70d97f35fb2-tmp\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.416297 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.416276 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7093f3f1-2b75-40b1-8148-f70d97f35fb2-webhook-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.416370 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.416316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7093f3f1-2b75-40b1-8148-f70d97f35fb2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.424302 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.424283 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vpk\" (UniqueName: \"kubernetes.io/projected/7093f3f1-2b75-40b1-8148-f70d97f35fb2-kube-api-access-55vpk\") pod \"opendatahub-operator-controller-manager-648c847c44-65nxm\" (UID: \"7093f3f1-2b75-40b1-8148-f70d97f35fb2\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.428800 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.428778 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt"] Sep 17 18:08:07.514451 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.514388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8d60341-160c-4382-8e7d-a5290f1a49b6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt\" (UID: \"e8d60341-160c-4382-8e7d-a5290f1a49b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" Sep 17 18:08:07.514451 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.514436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8d60341-160c-4382-8e7d-a5290f1a49b6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt\" (UID: \"e8d60341-160c-4382-8e7d-a5290f1a49b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" Sep 17 18:08:07.514603 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.514472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45pt9\" (UniqueName: \"kubernetes.io/projected/e8d60341-160c-4382-8e7d-a5290f1a49b6-kube-api-access-45pt9\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt\" (UID: \"e8d60341-160c-4382-8e7d-a5290f1a49b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" Sep 17 18:08:07.514742 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.514722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8d60341-160c-4382-8e7d-a5290f1a49b6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt\" (UID: \"e8d60341-160c-4382-8e7d-a5290f1a49b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" Sep 17 18:08:07.514803 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.514783 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8d60341-160c-4382-8e7d-a5290f1a49b6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt\" (UID: \"e8d60341-160c-4382-8e7d-a5290f1a49b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" Sep 17 18:08:07.523367 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.523346 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:07.526508 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.526483 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45pt9\" (UniqueName: \"kubernetes.io/projected/e8d60341-160c-4382-8e7d-a5290f1a49b6-kube-api-access-45pt9\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt\" (UID: \"e8d60341-160c-4382-8e7d-a5290f1a49b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" Sep 17 18:08:07.735105 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.735074 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" Sep 17 18:08:07.806542 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.806500 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm"] Sep 17 18:08:07.810412 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:08:07.810388 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7093f3f1_2b75_40b1_8148_f70d97f35fb2.slice/crio-29f6964dbe7c23d6abb3259bd3a357bc73114762f84ca449ea05c6242960e238 WatchSource:0}: Error finding container 29f6964dbe7c23d6abb3259bd3a357bc73114762f84ca449ea05c6242960e238: Status 404 returned error can't find the container with id 29f6964dbe7c23d6abb3259bd3a357bc73114762f84ca449ea05c6242960e238 Sep 17 18:08:07.812204 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.812185 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:08:07.923901 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:07.923876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" event={"ID":"7093f3f1-2b75-40b1-8148-f70d97f35fb2","Type":"ContainerStarted","Data":"29f6964dbe7c23d6abb3259bd3a357bc73114762f84ca449ea05c6242960e238"} Sep 17 18:08:08.023843 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:08.023789 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt"] Sep 17 18:08:08.025737 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:08:08.025718 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d60341_160c_4382_8e7d_a5290f1a49b6.slice/crio-ebc0ec40d59209d59df8a55c6eb808fe2b7fe9a3d58670fd20e128195d7806b4 WatchSource:0}: Error finding container ebc0ec40d59209d59df8a55c6eb808fe2b7fe9a3d58670fd20e128195d7806b4: Status 404 returned error can't find the container with id ebc0ec40d59209d59df8a55c6eb808fe2b7fe9a3d58670fd20e128195d7806b4 Sep 17 18:08:08.933826 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:08.933789 2564 generic.go:356] "Generic (PLEG): container finished" podID="e8d60341-160c-4382-8e7d-a5290f1a49b6" containerID="703cd74ed17f067a22612d4710b2b0931d5bad9efa2215f6d66d512e484378aa" exitCode=0 Sep 17 18:08:08.934271 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:08.933879 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" event={"ID":"e8d60341-160c-4382-8e7d-a5290f1a49b6","Type":"ContainerDied","Data":"703cd74ed17f067a22612d4710b2b0931d5bad9efa2215f6d66d512e484378aa"} Sep 17 18:08:08.934271 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:08.933997 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" event={"ID":"e8d60341-160c-4382-8e7d-a5290f1a49b6","Type":"ContainerStarted","Data":"ebc0ec40d59209d59df8a55c6eb808fe2b7fe9a3d58670fd20e128195d7806b4"} Sep 17 18:08:11.949039 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:11.948998 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" event={"ID":"7093f3f1-2b75-40b1-8148-f70d97f35fb2","Type":"ContainerStarted","Data":"a7fefc53598b76ca00bfc7d98200f6e7f2991252e126b7c60cc342b21175f486"} Sep 17 18:08:11.951246 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:11.951216 2564 generic.go:356] "Generic (PLEG): container finished" podID="e8d60341-160c-4382-8e7d-a5290f1a49b6" containerID="9123b1406de0e435a549f8541a8c82a08b815e9f55d89aa3640fab7fda2e16f6" exitCode=0 Sep 17 18:08:11.951364 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:11.951279 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" event={"ID":"e8d60341-160c-4382-8e7d-a5290f1a49b6","Type":"ContainerDied","Data":"9123b1406de0e435a549f8541a8c82a08b815e9f55d89aa3640fab7fda2e16f6"} Sep 17 18:08:12.957384 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:12.957344 2564 generic.go:356] "Generic (PLEG): container finished" podID="7093f3f1-2b75-40b1-8148-f70d97f35fb2" containerID="a7fefc53598b76ca00bfc7d98200f6e7f2991252e126b7c60cc342b21175f486" exitCode=0 Sep 17 18:08:12.957875 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:12.957438 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" event={"ID":"7093f3f1-2b75-40b1-8148-f70d97f35fb2","Type":"ContainerDied","Data":"a7fefc53598b76ca00bfc7d98200f6e7f2991252e126b7c60cc342b21175f486"} Sep 17 18:08:12.960451 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:12.960423 2564 generic.go:356] "Generic (PLEG): container finished" podID="e8d60341-160c-4382-8e7d-a5290f1a49b6" containerID="dbdea56ae3cf115c50638ea3fb86b1ca9ebbdfdc630da8381ef1fde56986fa71" exitCode=0 Sep 17 18:08:12.960616 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:12.960591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" event={"ID":"e8d60341-160c-4382-8e7d-a5290f1a49b6","Type":"ContainerDied","Data":"dbdea56ae3cf115c50638ea3fb86b1ca9ebbdfdc630da8381ef1fde56986fa71"} Sep 17 18:08:13.968332 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:13.968293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" event={"ID":"7093f3f1-2b75-40b1-8148-f70d97f35fb2","Type":"ContainerStarted","Data":"80da2d9648f467dead2440a10c2bcce821c9f3fdef22de330338d300cced9bd5"} Sep 17 18:08:14.003258 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.003199 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" podStartSLOduration=2.472928999 podStartE2EDuration="7.003176796s" totalImagesPullingTime="3.445130334s" totalInitContainerRuntime="1.085117463s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:14.000727729 +0000 UTC m=+319.010914866" watchObservedRunningTime="2026-09-17 18:08:14.003176796 +0000 UTC m=+319.013363933" Sep 17 18:08:14.092784 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.092757 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" Sep 17 18:08:14.174282 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.174260 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8d60341-160c-4382-8e7d-a5290f1a49b6-bundle\") pod \"e8d60341-160c-4382-8e7d-a5290f1a49b6\" (UID: \"e8d60341-160c-4382-8e7d-a5290f1a49b6\") " Sep 17 18:08:14.174395 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.174290 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8d60341-160c-4382-8e7d-a5290f1a49b6-util\") pod \"e8d60341-160c-4382-8e7d-a5290f1a49b6\" (UID: \"e8d60341-160c-4382-8e7d-a5290f1a49b6\") " Sep 17 18:08:14.174395 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.174319 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45pt9\" (UniqueName: \"kubernetes.io/projected/e8d60341-160c-4382-8e7d-a5290f1a49b6-kube-api-access-45pt9\") pod \"e8d60341-160c-4382-8e7d-a5290f1a49b6\" (UID: \"e8d60341-160c-4382-8e7d-a5290f1a49b6\") " Sep 17 18:08:14.174973 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.174910 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d60341-160c-4382-8e7d-a5290f1a49b6-bundle" (OuterVolumeSpecName: "bundle") pod "e8d60341-160c-4382-8e7d-a5290f1a49b6" (UID: "e8d60341-160c-4382-8e7d-a5290f1a49b6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:08:14.176447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.176426 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d60341-160c-4382-8e7d-a5290f1a49b6-kube-api-access-45pt9" (OuterVolumeSpecName: "kube-api-access-45pt9") pod "e8d60341-160c-4382-8e7d-a5290f1a49b6" (UID: "e8d60341-160c-4382-8e7d-a5290f1a49b6"). InnerVolumeSpecName "kube-api-access-45pt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:08:14.179185 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.179154 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d60341-160c-4382-8e7d-a5290f1a49b6-util" (OuterVolumeSpecName: "util") pod "e8d60341-160c-4382-8e7d-a5290f1a49b6" (UID: "e8d60341-160c-4382-8e7d-a5290f1a49b6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:08:14.275095 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.275036 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45pt9\" (UniqueName: \"kubernetes.io/projected/e8d60341-160c-4382-8e7d-a5290f1a49b6-kube-api-access-45pt9\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:08:14.275095 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.275067 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8d60341-160c-4382-8e7d-a5290f1a49b6-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:08:14.275095 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.275075 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8d60341-160c-4382-8e7d-a5290f1a49b6-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:08:14.975331 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.975301 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" Sep 17 18:08:14.975331 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.975323 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fqmbt" event={"ID":"e8d60341-160c-4382-8e7d-a5290f1a49b6","Type":"ContainerDied","Data":"ebc0ec40d59209d59df8a55c6eb808fe2b7fe9a3d58670fd20e128195d7806b4"} Sep 17 18:08:14.975898 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.975387 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebc0ec40d59209d59df8a55c6eb808fe2b7fe9a3d58670fd20e128195d7806b4" Sep 17 18:08:14.975898 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:14.975780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:25.982621 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:25.982590 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-65nxm" Sep 17 18:08:26.011042 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.011012 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg"] Sep 17 18:08:26.011316 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.011304 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d60341-160c-4382-8e7d-a5290f1a49b6" containerName="extract" Sep 17 18:08:26.011359 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.011318 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d60341-160c-4382-8e7d-a5290f1a49b6" containerName="extract" Sep 17 18:08:26.011359 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.011334 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d60341-160c-4382-8e7d-a5290f1a49b6" containerName="util" Sep 17 18:08:26.011359 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.011340 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d60341-160c-4382-8e7d-a5290f1a49b6" containerName="util" Sep 17 18:08:26.011359 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.011350 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d60341-160c-4382-8e7d-a5290f1a49b6" containerName="pull" Sep 17 18:08:26.011359 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.011357 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d60341-160c-4382-8e7d-a5290f1a49b6" containerName="pull" Sep 17 18:08:26.011581 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.011409 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8d60341-160c-4382-8e7d-a5290f1a49b6" containerName="extract" Sep 17 18:08:26.018560 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.018522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.023717 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.023699 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 17 18:08:26.023888 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.023869 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 17 18:08:26.023978 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.023959 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:08:26.024045 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.024000 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tmvdk\"" Sep 17 18:08:26.024492 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.024473 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 17 18:08:26.024594 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.024478 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 17 18:08:26.034220 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.034185 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg"] Sep 17 18:08:26.055029 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.054993 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c63ec5b2-c4f6-4900-aef5-0dac872ee565-cert\") pod \"lws-controller-manager-b8b8795dc-bx2tg\" (UID: \"c63ec5b2-c4f6-4900-aef5-0dac872ee565\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.055152 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.055105 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c63ec5b2-c4f6-4900-aef5-0dac872ee565-manager-config\") pod \"lws-controller-manager-b8b8795dc-bx2tg\" (UID: \"c63ec5b2-c4f6-4900-aef5-0dac872ee565\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.055152 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.055138 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c63ec5b2-c4f6-4900-aef5-0dac872ee565-metrics-cert\") pod \"lws-controller-manager-b8b8795dc-bx2tg\" (UID: \"c63ec5b2-c4f6-4900-aef5-0dac872ee565\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.055263 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.055166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmvmk\" (UniqueName: \"kubernetes.io/projected/c63ec5b2-c4f6-4900-aef5-0dac872ee565-kube-api-access-gmvmk\") pod \"lws-controller-manager-b8b8795dc-bx2tg\" (UID: \"c63ec5b2-c4f6-4900-aef5-0dac872ee565\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.156601 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.156573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c63ec5b2-c4f6-4900-aef5-0dac872ee565-cert\") pod \"lws-controller-manager-b8b8795dc-bx2tg\" (UID: \"c63ec5b2-c4f6-4900-aef5-0dac872ee565\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.156759 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.156639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c63ec5b2-c4f6-4900-aef5-0dac872ee565-manager-config\") pod \"lws-controller-manager-b8b8795dc-bx2tg\" (UID: \"c63ec5b2-c4f6-4900-aef5-0dac872ee565\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.156759 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.156664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c63ec5b2-c4f6-4900-aef5-0dac872ee565-metrics-cert\") pod \"lws-controller-manager-b8b8795dc-bx2tg\" (UID: \"c63ec5b2-c4f6-4900-aef5-0dac872ee565\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.156759 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.156680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmvmk\" (UniqueName: \"kubernetes.io/projected/c63ec5b2-c4f6-4900-aef5-0dac872ee565-kube-api-access-gmvmk\") pod \"lws-controller-manager-b8b8795dc-bx2tg\" (UID: \"c63ec5b2-c4f6-4900-aef5-0dac872ee565\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.157186 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.157163 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c63ec5b2-c4f6-4900-aef5-0dac872ee565-manager-config\") pod \"lws-controller-manager-b8b8795dc-bx2tg\" (UID: \"c63ec5b2-c4f6-4900-aef5-0dac872ee565\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.159304 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.159280 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c63ec5b2-c4f6-4900-aef5-0dac872ee565-cert\") pod \"lws-controller-manager-b8b8795dc-bx2tg\" (UID: \"c63ec5b2-c4f6-4900-aef5-0dac872ee565\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.159383 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.159287 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c63ec5b2-c4f6-4900-aef5-0dac872ee565-metrics-cert\") pod \"lws-controller-manager-b8b8795dc-bx2tg\" (UID: \"c63ec5b2-c4f6-4900-aef5-0dac872ee565\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.167477 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.167458 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmvmk\" (UniqueName: \"kubernetes.io/projected/c63ec5b2-c4f6-4900-aef5-0dac872ee565-kube-api-access-gmvmk\") pod \"lws-controller-manager-b8b8795dc-bx2tg\" (UID: \"c63ec5b2-c4f6-4900-aef5-0dac872ee565\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.327780 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.327759 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:26.614080 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:26.614057 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg"] Sep 17 18:08:26.616030 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:08:26.615996 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63ec5b2_c4f6_4900_aef5_0dac872ee565.slice/crio-948e352bf7fac7e77e18320612ce472052362b90b931a2e2343d51fda34f56ed WatchSource:0}: Error finding container 948e352bf7fac7e77e18320612ce472052362b90b931a2e2343d51fda34f56ed: Status 404 returned error can't find the container with id 948e352bf7fac7e77e18320612ce472052362b90b931a2e2343d51fda34f56ed Sep 17 18:08:27.046344 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:27.046274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" event={"ID":"c63ec5b2-c4f6-4900-aef5-0dac872ee565","Type":"ContainerStarted","Data":"948e352bf7fac7e77e18320612ce472052362b90b931a2e2343d51fda34f56ed"} Sep 17 18:08:29.059321 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:29.059273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" event={"ID":"c63ec5b2-c4f6-4900-aef5-0dac872ee565","Type":"ContainerStarted","Data":"f35ed81fa6f33b0a5f97caee66ea8381f01aacedf6ac8fd48ff124e1f759bfdc"} Sep 17 18:08:29.059826 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:29.059347 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:29.093803 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:29.093754 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" podStartSLOduration=2.676202523 podStartE2EDuration="4.093737213s" totalImagesPullingTime="1.41753469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:29.092273768 +0000 UTC m=+334.102460903" watchObservedRunningTime="2026-09-17 18:08:29.093737213 +0000 UTC m=+334.103924349" Sep 17 18:08:36.281196 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.281161 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw"] Sep 17 18:08:36.284600 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.284583 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.288948 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.288925 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-l2krp\"" Sep 17 18:08:36.289087 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.288955 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 18:08:36.334768 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.334740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74a9357-78d5-4afb-babe-31e5a16973b5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.334879 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.334771 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74a9357-78d5-4afb-babe-31e5a16973b5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.334879 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.334748 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw"] Sep 17 18:08:36.334879 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.334805 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74a9357-78d5-4afb-babe-31e5a16973b5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.334879 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.334820 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.334879 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.334855 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.334879 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.334872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.335089 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.334888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.335089 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.334905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.335089 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.334919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nh2x\" (UniqueName: \"kubernetes.io/projected/e74a9357-78d5-4afb-babe-31e5a16973b5-kube-api-access-9nh2x\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.435616 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.435589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74a9357-78d5-4afb-babe-31e5a16973b5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.435725 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.435618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74a9357-78d5-4afb-babe-31e5a16973b5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.435725 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.435641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74a9357-78d5-4afb-babe-31e5a16973b5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.435725 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.435658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.435725 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.435686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.435725 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.435706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.435943 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.435735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.435943 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.435766 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.435943 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.435788 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nh2x\" (UniqueName: \"kubernetes.io/projected/e74a9357-78d5-4afb-babe-31e5a16973b5-kube-api-access-9nh2x\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.436174 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.436146 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.436289 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.436271 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.436361 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.436294 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.436361 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.436340 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.437053 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.437025 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74a9357-78d5-4afb-babe-31e5a16973b5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.438756 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.438733 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74a9357-78d5-4afb-babe-31e5a16973b5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.443424 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.443404 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74a9357-78d5-4afb-babe-31e5a16973b5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.452921 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.452897 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74a9357-78d5-4afb-babe-31e5a16973b5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.453011 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.452995 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nh2x\" (UniqueName: \"kubernetes.io/projected/e74a9357-78d5-4afb-babe-31e5a16973b5-kube-api-access-9nh2x\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw\" (UID: \"e74a9357-78d5-4afb-babe-31e5a16973b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.594927 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.594903 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:36.881619 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:36.881595 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw"] Sep 17 18:08:36.883656 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:08:36.883629 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74a9357_78d5_4afb_babe_31e5a16973b5.slice/crio-bab657e4fc5577976d4795d95a1cf947da9d7f303fa9a2dd1ac0a4b749caaf66 WatchSource:0}: Error finding container bab657e4fc5577976d4795d95a1cf947da9d7f303fa9a2dd1ac0a4b749caaf66: Status 404 returned error can't find the container with id bab657e4fc5577976d4795d95a1cf947da9d7f303fa9a2dd1ac0a4b749caaf66 Sep 17 18:08:37.096501 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:37.096469 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" event={"ID":"e74a9357-78d5-4afb-babe-31e5a16973b5","Type":"ContainerStarted","Data":"bab657e4fc5577976d4795d95a1cf947da9d7f303fa9a2dd1ac0a4b749caaf66"} Sep 17 18:08:38.758219 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:38.758183 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vb689"] Sep 17 18:08:38.764726 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:38.764697 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vb689" Sep 17 18:08:38.767432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:38.767399 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 17 18:08:38.768034 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:38.768016 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-fr6w5\"" Sep 17 18:08:38.774304 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:38.774278 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vb689"] Sep 17 18:08:38.857270 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:38.857231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqm7z\" (UniqueName: \"kubernetes.io/projected/0df7fffc-37a3-46eb-9767-455a50dfc379-kube-api-access-tqm7z\") pod \"odh-model-controller-7998c67dc4-vb689\" (UID: \"0df7fffc-37a3-46eb-9767-455a50dfc379\") " pod="opendatahub/odh-model-controller-7998c67dc4-vb689" Sep 17 18:08:38.857433 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:38.857289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0df7fffc-37a3-46eb-9767-455a50dfc379-cert\") pod \"odh-model-controller-7998c67dc4-vb689\" (UID: \"0df7fffc-37a3-46eb-9767-455a50dfc379\") " pod="opendatahub/odh-model-controller-7998c67dc4-vb689" Sep 17 18:08:38.957991 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:38.957958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqm7z\" (UniqueName: \"kubernetes.io/projected/0df7fffc-37a3-46eb-9767-455a50dfc379-kube-api-access-tqm7z\") pod \"odh-model-controller-7998c67dc4-vb689\" (UID: \"0df7fffc-37a3-46eb-9767-455a50dfc379\") " pod="opendatahub/odh-model-controller-7998c67dc4-vb689" Sep 17 18:08:38.958178 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:38.958026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0df7fffc-37a3-46eb-9767-455a50dfc379-cert\") pod \"odh-model-controller-7998c67dc4-vb689\" (UID: \"0df7fffc-37a3-46eb-9767-455a50dfc379\") " pod="opendatahub/odh-model-controller-7998c67dc4-vb689" Sep 17 18:08:38.958178 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:08:38.958147 2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 17 18:08:38.958308 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:08:38.958214 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0df7fffc-37a3-46eb-9767-455a50dfc379-cert podName:0df7fffc-37a3-46eb-9767-455a50dfc379 nodeName:}" failed. No retries permitted until 2026-09-17 18:08:39.458193481 +0000 UTC m=+344.468380598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0df7fffc-37a3-46eb-9767-455a50dfc379-cert") pod "odh-model-controller-7998c67dc4-vb689" (UID: "0df7fffc-37a3-46eb-9767-455a50dfc379") : secret "odh-model-controller-webhook-cert" not found Sep 17 18:08:38.973649 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:38.973623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqm7z\" (UniqueName: \"kubernetes.io/projected/0df7fffc-37a3-46eb-9767-455a50dfc379-kube-api-access-tqm7z\") pod \"odh-model-controller-7998c67dc4-vb689\" (UID: \"0df7fffc-37a3-46eb-9767-455a50dfc379\") " pod="opendatahub/odh-model-controller-7998c67dc4-vb689" Sep 17 18:08:39.442367 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.442339 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-gb5lj"] Sep 17 18:08:39.447959 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.447926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-gb5lj" Sep 17 18:08:39.450692 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.450666 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 17 18:08:39.461933 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.461910 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-gb5lj"] Sep 17 18:08:39.462751 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.462725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0df7fffc-37a3-46eb-9767-455a50dfc379-cert\") pod \"odh-model-controller-7998c67dc4-vb689\" (UID: \"0df7fffc-37a3-46eb-9767-455a50dfc379\") " pod="opendatahub/odh-model-controller-7998c67dc4-vb689" Sep 17 18:08:39.466283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.466262 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0df7fffc-37a3-46eb-9767-455a50dfc379-cert\") pod \"odh-model-controller-7998c67dc4-vb689\" (UID: \"0df7fffc-37a3-46eb-9767-455a50dfc379\") " pod="opendatahub/odh-model-controller-7998c67dc4-vb689" Sep 17 18:08:39.564284 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.564251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6shqj\" (UniqueName: \"kubernetes.io/projected/d30ba015-e452-4a82-a227-7bf26d504a93-kube-api-access-6shqj\") pod \"model-serving-api-54d89bd5fc-gb5lj\" (UID: \"d30ba015-e452-4a82-a227-7bf26d504a93\") " pod="opendatahub/model-serving-api-54d89bd5fc-gb5lj" Sep 17 18:08:39.564445 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.564348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30ba015-e452-4a82-a227-7bf26d504a93-tls-certs\") pod \"model-serving-api-54d89bd5fc-gb5lj\" (UID: \"d30ba015-e452-4a82-a227-7bf26d504a93\") " pod="opendatahub/model-serving-api-54d89bd5fc-gb5lj" Sep 17 18:08:39.665206 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.665169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6shqj\" (UniqueName: \"kubernetes.io/projected/d30ba015-e452-4a82-a227-7bf26d504a93-kube-api-access-6shqj\") pod \"model-serving-api-54d89bd5fc-gb5lj\" (UID: \"d30ba015-e452-4a82-a227-7bf26d504a93\") " pod="opendatahub/model-serving-api-54d89bd5fc-gb5lj" Sep 17 18:08:39.665375 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.665237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30ba015-e452-4a82-a227-7bf26d504a93-tls-certs\") pod \"model-serving-api-54d89bd5fc-gb5lj\" (UID: \"d30ba015-e452-4a82-a227-7bf26d504a93\") " pod="opendatahub/model-serving-api-54d89bd5fc-gb5lj" Sep 17 18:08:39.667775 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.667749 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30ba015-e452-4a82-a227-7bf26d504a93-tls-certs\") pod \"model-serving-api-54d89bd5fc-gb5lj\" (UID: \"d30ba015-e452-4a82-a227-7bf26d504a93\") " pod="opendatahub/model-serving-api-54d89bd5fc-gb5lj" Sep 17 18:08:39.677513 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.677476 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vb689" Sep 17 18:08:39.680677 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.680413 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6shqj\" (UniqueName: \"kubernetes.io/projected/d30ba015-e452-4a82-a227-7bf26d504a93-kube-api-access-6shqj\") pod \"model-serving-api-54d89bd5fc-gb5lj\" (UID: \"d30ba015-e452-4a82-a227-7bf26d504a93\") " pod="opendatahub/model-serving-api-54d89bd5fc-gb5lj" Sep 17 18:08:39.737476 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.737424 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:08:39.737685 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.737615 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:08:39.737750 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.737696 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:08:39.762276 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.762050 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-gb5lj" Sep 17 18:08:39.973723 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:39.973685 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vb689"] Sep 17 18:08:39.973842 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:08:39.973826 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df7fffc_37a3_46eb_9767_455a50dfc379.slice/crio-f50745d1143603e7c294b666da802b96b37010e39503dfa83c1948a32b4251c2 WatchSource:0}: Error finding container f50745d1143603e7c294b666da802b96b37010e39503dfa83c1948a32b4251c2: Status 404 returned error can't find the container with id f50745d1143603e7c294b666da802b96b37010e39503dfa83c1948a32b4251c2 Sep 17 18:08:40.041315 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:40.041234 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-gb5lj"] Sep 17 18:08:40.045899 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:08:40.045874 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30ba015_e452_4a82_a227_7bf26d504a93.slice/crio-fd87bc8e158b48a1e8f79bac8b4edc59a5e19f3632bd326f53bc9a03e8d78123 WatchSource:0}: Error finding container fd87bc8e158b48a1e8f79bac8b4edc59a5e19f3632bd326f53bc9a03e8d78123: Status 404 returned error can't find the container with id fd87bc8e158b48a1e8f79bac8b4edc59a5e19f3632bd326f53bc9a03e8d78123 Sep 17 18:08:40.066893 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:40.066873 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-bx2tg" Sep 17 18:08:40.114166 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:40.114139 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" event={"ID":"e74a9357-78d5-4afb-babe-31e5a16973b5","Type":"ContainerStarted","Data":"b92c11a430d194a363f133b44ef48c78a1f9de06387e7891883381607703dcc0"} Sep 17 18:08:40.115546 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:40.115510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-gb5lj" event={"ID":"d30ba015-e452-4a82-a227-7bf26d504a93","Type":"ContainerStarted","Data":"fd87bc8e158b48a1e8f79bac8b4edc59a5e19f3632bd326f53bc9a03e8d78123"} Sep 17 18:08:40.116764 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:40.116742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vb689" event={"ID":"0df7fffc-37a3-46eb-9767-455a50dfc379","Type":"ContainerStarted","Data":"f50745d1143603e7c294b666da802b96b37010e39503dfa83c1948a32b4251c2"} Sep 17 18:08:40.150428 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:40.150387 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" podStartSLOduration=1.298737977 podStartE2EDuration="4.150372742s" totalImagesPullingTime="2.851634765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:40.150118949 +0000 UTC m=+345.160306085" watchObservedRunningTime="2026-09-17 18:08:40.150372742 +0000 UTC m=+345.160559924" Sep 17 18:08:40.595140 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:40.595112 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:40.600458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:40.600435 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:41.125641 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:41.125556 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:41.127107 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:41.127021 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw" Sep 17 18:08:44.142203 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:44.142167 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vb689" event={"ID":"0df7fffc-37a3-46eb-9767-455a50dfc379","Type":"ContainerStarted","Data":"97915bae32269481f517e547e27284e5c45497f18155a807d5618ab5563b653a"} Sep 17 18:08:44.142643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:44.142278 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-vb689" Sep 17 18:08:44.143916 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:44.143889 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-gb5lj" event={"ID":"d30ba015-e452-4a82-a227-7bf26d504a93","Type":"ContainerStarted","Data":"ed3ba3000ccb9648e681cec6a00b5bb5da34643186ea6c2bb4724d3726f7b937"} Sep 17 18:08:44.144026 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:44.144009 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-gb5lj" Sep 17 18:08:44.184208 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:44.184160 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-vb689" podStartSLOduration=2.14034764 podStartE2EDuration="6.184141641s" totalImagesPullingTime="4.043794001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:44.182909097 +0000 UTC m=+349.193096244" watchObservedRunningTime="2026-09-17 18:08:44.184141641 +0000 UTC m=+349.194328777" Sep 17 18:08:44.230900 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:44.230859 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-gb5lj" podStartSLOduration=1.220876451 podStartE2EDuration="5.230844716s" totalImagesPullingTime="4.009968265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:44.224238684 +0000 UTC m=+349.234425819" watchObservedRunningTime="2026-09-17 18:08:44.230844716 +0000 UTC m=+349.241031852" Sep 17 18:08:46.378399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.378367 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf"] Sep 17 18:08:46.384740 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.383124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.387014 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.386993 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 17 18:08:46.387387 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.387369 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 17 18:08:46.397413 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.397394 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 17 18:08:46.397801 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.397787 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-474z2\"" Sep 17 18:08:46.404944 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.404924 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf"] Sep 17 18:08:46.522773 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.522740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49f4b11-b236-44a4-9078-b43c79561b83-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.522885 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.522795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e49f4b11-b236-44a4-9078-b43c79561b83-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.522885 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.522842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e49f4b11-b236-44a4-9078-b43c79561b83-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.522968 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.522891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vnt\" (UniqueName: \"kubernetes.io/projected/e49f4b11-b236-44a4-9078-b43c79561b83-kube-api-access-w7vnt\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.522968 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.522921 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e49f4b11-b236-44a4-9078-b43c79561b83-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.522968 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.522957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e49f4b11-b236-44a4-9078-b43c79561b83-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.523086 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.523003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49f4b11-b236-44a4-9078-b43c79561b83-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.624288 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.624237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e49f4b11-b236-44a4-9078-b43c79561b83-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.624288 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.624276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e49f4b11-b236-44a4-9078-b43c79561b83-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.624467 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.624328 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vnt\" (UniqueName: \"kubernetes.io/projected/e49f4b11-b236-44a4-9078-b43c79561b83-kube-api-access-w7vnt\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.624467 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.624354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e49f4b11-b236-44a4-9078-b43c79561b83-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.624467 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.624395 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e49f4b11-b236-44a4-9078-b43c79561b83-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.624467 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.624426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49f4b11-b236-44a4-9078-b43c79561b83-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.624467 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.624456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49f4b11-b236-44a4-9078-b43c79561b83-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.625072 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.625039 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e49f4b11-b236-44a4-9078-b43c79561b83-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.627659 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.627629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49f4b11-b236-44a4-9078-b43c79561b83-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.627788 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.627638 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e49f4b11-b236-44a4-9078-b43c79561b83-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.627788 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.627723 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e49f4b11-b236-44a4-9078-b43c79561b83-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.627873 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.627787 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e49f4b11-b236-44a4-9078-b43c79561b83-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.655022 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.654790 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vnt\" (UniqueName: \"kubernetes.io/projected/e49f4b11-b236-44a4-9078-b43c79561b83-kube-api-access-w7vnt\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.655701 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.655677 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49f4b11-b236-44a4-9078-b43c79561b83-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cmpkf\" (UID: \"e49f4b11-b236-44a4-9078-b43c79561b83\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.693357 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.693327 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:46.985787 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:46.985759 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf"] Sep 17 18:08:46.987232 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:08:46.987204 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49f4b11_b236_44a4_9078_b43c79561b83.slice/crio-6fbc23e366f771838f661defd00518dcef172cae500f780e94429442ab3610dd WatchSource:0}: Error finding container 6fbc23e366f771838f661defd00518dcef172cae500f780e94429442ab3610dd: Status 404 returned error can't find the container with id 6fbc23e366f771838f661defd00518dcef172cae500f780e94429442ab3610dd Sep 17 18:08:47.160396 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:47.160362 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" event={"ID":"e49f4b11-b236-44a4-9078-b43c79561b83","Type":"ContainerStarted","Data":"6fbc23e366f771838f661defd00518dcef172cae500f780e94429442ab3610dd"} Sep 17 18:08:49.324624 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:49.324585 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:08:49.324990 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:49.324679 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:08:50.182498 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:50.182466 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" event={"ID":"e49f4b11-b236-44a4-9078-b43c79561b83","Type":"ContainerStarted","Data":"0cf37eb789d671d96d4d8ab50b6658c171ce272a12753cd9443f1f710cf7ebe9"} Sep 17 18:08:50.182704 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:50.182656 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:50.184827 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:50.184798 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-cmpkf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 18:08:50.184969 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:50.184854 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" podUID="e49f4b11-b236-44a4-9078-b43c79561b83" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:08:50.255932 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:50.255876 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" podStartSLOduration=1.920923532 podStartE2EDuration="4.255855844s" totalImagesPullingTime="2.334932312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:50.236162859 +0000 UTC m=+355.246349996" watchObservedRunningTime="2026-09-17 18:08:50.255855844 +0000 UTC m=+355.266042981" Sep 17 18:08:51.188161 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:51.188132 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cmpkf" Sep 17 18:08:55.151147 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:55.151118 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-vb689" Sep 17 18:08:55.153434 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:08:55.153408 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-gb5lj" Sep 17 18:09:44.448468 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.448430 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf"] Sep 17 18:09:44.453744 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.453719 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" Sep 17 18:09:44.456544 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.456510 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vwsz8\"" Sep 17 18:09:44.456824 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.456808 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 18:09:44.457282 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.457266 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 18:09:44.462488 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.462464 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf"] Sep 17 18:09:44.541598 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.541573 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj"] Sep 17 18:09:44.543951 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.543937 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" Sep 17 18:09:44.554802 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.554782 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj"] Sep 17 18:09:44.556399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.556377 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf\" (UID: \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" Sep 17 18:09:44.556503 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.556438 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf\" (UID: \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" Sep 17 18:09:44.556574 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.556508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m54m7\" (UniqueName: \"kubernetes.io/projected/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-kube-api-access-m54m7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf\" (UID: \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" Sep 17 18:09:44.650019 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.649992 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg"] Sep 17 18:09:44.652581 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.652565 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" Sep 17 18:09:44.657432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.657408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e903e93-9c1b-41be-94b1-71fefb67c52e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj\" (UID: \"2e903e93-9c1b-41be-94b1-71fefb67c52e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" Sep 17 18:09:44.657571 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.657443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e903e93-9c1b-41be-94b1-71fefb67c52e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj\" (UID: \"2e903e93-9c1b-41be-94b1-71fefb67c52e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" Sep 17 18:09:44.657571 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.657477 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zj48\" (UniqueName: \"kubernetes.io/projected/2e903e93-9c1b-41be-94b1-71fefb67c52e-kube-api-access-8zj48\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj\" (UID: \"2e903e93-9c1b-41be-94b1-71fefb67c52e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" Sep 17 18:09:44.657571 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.657515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf\" (UID: \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" Sep 17 18:09:44.657736 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.657577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m54m7\" (UniqueName: \"kubernetes.io/projected/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-kube-api-access-m54m7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf\" (UID: \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" Sep 17 18:09:44.657736 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.657662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf\" (UID: \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" Sep 17 18:09:44.657971 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.657951 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf\" (UID: \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" Sep 17 18:09:44.658033 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.657979 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf\" (UID: \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" Sep 17 18:09:44.662988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.662967 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg"] Sep 17 18:09:44.666230 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.666208 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m54m7\" (UniqueName: \"kubernetes.io/projected/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-kube-api-access-m54m7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf\" (UID: \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" Sep 17 18:09:44.744554 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.744484 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb"] Sep 17 18:09:44.746939 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.746925 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" Sep 17 18:09:44.758302 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.758283 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e903e93-9c1b-41be-94b1-71fefb67c52e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj\" (UID: \"2e903e93-9c1b-41be-94b1-71fefb67c52e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" Sep 17 18:09:44.758385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.758307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e903e93-9c1b-41be-94b1-71fefb67c52e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj\" (UID: \"2e903e93-9c1b-41be-94b1-71fefb67c52e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" Sep 17 18:09:44.758385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.758326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zj48\" (UniqueName: \"kubernetes.io/projected/2e903e93-9c1b-41be-94b1-71fefb67c52e-kube-api-access-8zj48\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj\" (UID: \"2e903e93-9c1b-41be-94b1-71fefb67c52e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" Sep 17 18:09:44.758385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.758351 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z4sx\" (UniqueName: \"kubernetes.io/projected/3c2262c4-3f17-4ff5-be69-b42754e3ce96-kube-api-access-9z4sx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg\" (UID: \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" Sep 17 18:09:44.758509 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.758398 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c2262c4-3f17-4ff5-be69-b42754e3ce96-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg\" (UID: \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" Sep 17 18:09:44.758509 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.758420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c2262c4-3f17-4ff5-be69-b42754e3ce96-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg\" (UID: \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" Sep 17 18:09:44.758689 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.758667 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e903e93-9c1b-41be-94b1-71fefb67c52e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj\" (UID: \"2e903e93-9c1b-41be-94b1-71fefb67c52e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" Sep 17 18:09:44.758726 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.758688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e903e93-9c1b-41be-94b1-71fefb67c52e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj\" (UID: \"2e903e93-9c1b-41be-94b1-71fefb67c52e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" Sep 17 18:09:44.762161 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.762142 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb"] Sep 17 18:09:44.764176 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.764158 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" Sep 17 18:09:44.769612 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.769593 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zj48\" (UniqueName: \"kubernetes.io/projected/2e903e93-9c1b-41be-94b1-71fefb67c52e-kube-api-access-8zj48\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj\" (UID: \"2e903e93-9c1b-41be-94b1-71fefb67c52e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" Sep 17 18:09:44.851958 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.851924 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" Sep 17 18:09:44.859842 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.859820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c2262c4-3f17-4ff5-be69-b42754e3ce96-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg\" (UID: \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" Sep 17 18:09:44.859957 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.859848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c2262c4-3f17-4ff5-be69-b42754e3ce96-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg\" (UID: \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" Sep 17 18:09:44.859957 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.859896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43e36e9a-e2ac-4557-9a71-02bba2b20550-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb\" (UID: \"43e36e9a-e2ac-4557-9a71-02bba2b20550\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" Sep 17 18:09:44.859957 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.859936 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzj2f\" (UniqueName: \"kubernetes.io/projected/43e36e9a-e2ac-4557-9a71-02bba2b20550-kube-api-access-tzj2f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb\" (UID: \"43e36e9a-e2ac-4557-9a71-02bba2b20550\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" Sep 17 18:09:44.860108 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.860029 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z4sx\" (UniqueName: \"kubernetes.io/projected/3c2262c4-3f17-4ff5-be69-b42754e3ce96-kube-api-access-9z4sx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg\" (UID: \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" Sep 17 18:09:44.860108 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.860061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43e36e9a-e2ac-4557-9a71-02bba2b20550-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb\" (UID: \"43e36e9a-e2ac-4557-9a71-02bba2b20550\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" Sep 17 18:09:44.860259 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.860235 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c2262c4-3f17-4ff5-be69-b42754e3ce96-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg\" (UID: \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" Sep 17 18:09:44.860317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.860239 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c2262c4-3f17-4ff5-be69-b42754e3ce96-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg\" (UID: \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" Sep 17 18:09:44.868956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.868932 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z4sx\" (UniqueName: \"kubernetes.io/projected/3c2262c4-3f17-4ff5-be69-b42754e3ce96-kube-api-access-9z4sx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg\" (UID: \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" Sep 17 18:09:44.961068 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.961042 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" Sep 17 18:09:44.961287 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.961266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43e36e9a-e2ac-4557-9a71-02bba2b20550-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb\" (UID: \"43e36e9a-e2ac-4557-9a71-02bba2b20550\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" Sep 17 18:09:44.961353 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.961313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43e36e9a-e2ac-4557-9a71-02bba2b20550-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb\" (UID: \"43e36e9a-e2ac-4557-9a71-02bba2b20550\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" Sep 17 18:09:44.961353 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.961339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzj2f\" (UniqueName: \"kubernetes.io/projected/43e36e9a-e2ac-4557-9a71-02bba2b20550-kube-api-access-tzj2f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb\" (UID: \"43e36e9a-e2ac-4557-9a71-02bba2b20550\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" Sep 17 18:09:44.961708 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.961688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43e36e9a-e2ac-4557-9a71-02bba2b20550-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb\" (UID: \"43e36e9a-e2ac-4557-9a71-02bba2b20550\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" Sep 17 18:09:44.961792 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.961766 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43e36e9a-e2ac-4557-9a71-02bba2b20550-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb\" (UID: \"43e36e9a-e2ac-4557-9a71-02bba2b20550\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" Sep 17 18:09:44.970981 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:44.970956 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzj2f\" (UniqueName: \"kubernetes.io/projected/43e36e9a-e2ac-4557-9a71-02bba2b20550-kube-api-access-tzj2f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb\" (UID: \"43e36e9a-e2ac-4557-9a71-02bba2b20550\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" Sep 17 18:09:45.042136 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.042107 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf"] Sep 17 18:09:45.042954 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:09:45.042922 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c363b98_eadf_468d_b0ba_a60e5ad25e4b.slice/crio-5c1d532c195a6272a5cbf454b3ad1b82ae19b1d14a8a4dcd4442806f5bcae3b5 WatchSource:0}: Error finding container 5c1d532c195a6272a5cbf454b3ad1b82ae19b1d14a8a4dcd4442806f5bcae3b5: Status 404 returned error can't find the container with id 5c1d532c195a6272a5cbf454b3ad1b82ae19b1d14a8a4dcd4442806f5bcae3b5 Sep 17 18:09:45.055857 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.055831 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" Sep 17 18:09:45.146829 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.142049 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj"] Sep 17 18:09:45.238714 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.238689 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg"] Sep 17 18:09:45.239546 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:09:45.239501 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2262c4_3f17_4ff5_be69_b42754e3ce96.slice/crio-aacae33726b3ee1ca8d9e01a340fcb1901ad6c899b4ee2e2e82eaaf1d8ba9b02 WatchSource:0}: Error finding container aacae33726b3ee1ca8d9e01a340fcb1901ad6c899b4ee2e2e82eaaf1d8ba9b02: Status 404 returned error can't find the container with id aacae33726b3ee1ca8d9e01a340fcb1901ad6c899b4ee2e2e82eaaf1d8ba9b02 Sep 17 18:09:45.335083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.335054 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb"] Sep 17 18:09:45.367269 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:09:45.367236 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e36e9a_e2ac_4557_9a71_02bba2b20550.slice/crio-a160e3ececa078df868e4773b0e8cd2f972f2e4a98310ec2f20ff97d65fe8289 WatchSource:0}: Error finding container a160e3ececa078df868e4773b0e8cd2f972f2e4a98310ec2f20ff97d65fe8289: Status 404 returned error can't find the container with id a160e3ececa078df868e4773b0e8cd2f972f2e4a98310ec2f20ff97d65fe8289 Sep 17 18:09:45.457185 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.457155 2564 generic.go:356] "Generic (PLEG): container finished" podID="2e903e93-9c1b-41be-94b1-71fefb67c52e" containerID="1431a97392c42a6e6e910b9dd48df96c51045796e2b6c87baf95fe7166bbb096" exitCode=0 Sep 17 18:09:45.457505 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.457185 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" event={"ID":"2e903e93-9c1b-41be-94b1-71fefb67c52e","Type":"ContainerDied","Data":"1431a97392c42a6e6e910b9dd48df96c51045796e2b6c87baf95fe7166bbb096"} Sep 17 18:09:45.457505 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.457230 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" event={"ID":"2e903e93-9c1b-41be-94b1-71fefb67c52e","Type":"ContainerStarted","Data":"70e22216ff224bf2d7906c38e83107ef37f91422347142544e1494215b24a956"} Sep 17 18:09:45.459292 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.459271 2564 generic.go:356] "Generic (PLEG): container finished" podID="3c2262c4-3f17-4ff5-be69-b42754e3ce96" containerID="71d6a23066027321bc54dd6fe547f4076b9b1405f6893bf8b37c0c35a11c05da" exitCode=0 Sep 17 18:09:45.459382 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.459355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" event={"ID":"3c2262c4-3f17-4ff5-be69-b42754e3ce96","Type":"ContainerDied","Data":"71d6a23066027321bc54dd6fe547f4076b9b1405f6893bf8b37c0c35a11c05da"} Sep 17 18:09:45.459478 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.459411 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" event={"ID":"3c2262c4-3f17-4ff5-be69-b42754e3ce96","Type":"ContainerStarted","Data":"aacae33726b3ee1ca8d9e01a340fcb1901ad6c899b4ee2e2e82eaaf1d8ba9b02"} Sep 17 18:09:45.461355 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.461330 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c363b98-eadf-468d-b0ba-a60e5ad25e4b" containerID="76ca4078f4459f05c668d0b581d3f4b730e3526029e20c4d6573b0a769cef526" exitCode=0 Sep 17 18:09:45.461443 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.461415 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" event={"ID":"7c363b98-eadf-468d-b0ba-a60e5ad25e4b","Type":"ContainerDied","Data":"76ca4078f4459f05c668d0b581d3f4b730e3526029e20c4d6573b0a769cef526"} Sep 17 18:09:45.461504 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.461458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" event={"ID":"7c363b98-eadf-468d-b0ba-a60e5ad25e4b","Type":"ContainerStarted","Data":"5c1d532c195a6272a5cbf454b3ad1b82ae19b1d14a8a4dcd4442806f5bcae3b5"} Sep 17 18:09:45.463165 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.463144 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" event={"ID":"43e36e9a-e2ac-4557-9a71-02bba2b20550","Type":"ContainerStarted","Data":"e7c5d07db3cb585266c62c305b393a4cf1cf036958d1167ec38aee64c12c5fb5"} Sep 17 18:09:45.463257 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:45.463177 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" event={"ID":"43e36e9a-e2ac-4557-9a71-02bba2b20550","Type":"ContainerStarted","Data":"a160e3ececa078df868e4773b0e8cd2f972f2e4a98310ec2f20ff97d65fe8289"} Sep 17 18:09:46.470568 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:46.470518 2564 generic.go:356] "Generic (PLEG): container finished" podID="43e36e9a-e2ac-4557-9a71-02bba2b20550" containerID="e7c5d07db3cb585266c62c305b393a4cf1cf036958d1167ec38aee64c12c5fb5" exitCode=0 Sep 17 18:09:46.470835 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:46.470594 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" event={"ID":"43e36e9a-e2ac-4557-9a71-02bba2b20550","Type":"ContainerDied","Data":"e7c5d07db3cb585266c62c305b393a4cf1cf036958d1167ec38aee64c12c5fb5"} Sep 17 18:09:46.472287 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:46.472270 2564 generic.go:356] "Generic (PLEG): container finished" podID="2e903e93-9c1b-41be-94b1-71fefb67c52e" containerID="1d7d92d9a49a58bd43120bf8cba9b0a00b1c33cf362a8b5d75b3167b8692ee8e" exitCode=0 Sep 17 18:09:46.472374 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:46.472305 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" event={"ID":"2e903e93-9c1b-41be-94b1-71fefb67c52e","Type":"ContainerDied","Data":"1d7d92d9a49a58bd43120bf8cba9b0a00b1c33cf362a8b5d75b3167b8692ee8e"} Sep 17 18:09:47.482130 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:47.482093 2564 generic.go:356] "Generic (PLEG): container finished" podID="43e36e9a-e2ac-4557-9a71-02bba2b20550" containerID="e18f3a97ca7bd267af52a38a1fd3689b42cd4faacd8017c24a526064bef8eb84" exitCode=0 Sep 17 18:09:47.482557 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:47.482171 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" event={"ID":"43e36e9a-e2ac-4557-9a71-02bba2b20550","Type":"ContainerDied","Data":"e18f3a97ca7bd267af52a38a1fd3689b42cd4faacd8017c24a526064bef8eb84"} Sep 17 18:09:47.484567 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:47.484544 2564 generic.go:356] "Generic (PLEG): container finished" podID="2e903e93-9c1b-41be-94b1-71fefb67c52e" containerID="9b1a5d16d8e8789990a1c40930be8be8d3a33560866a0caaf05cdf5bf7149933" exitCode=0 Sep 17 18:09:47.484755 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:47.484630 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" event={"ID":"2e903e93-9c1b-41be-94b1-71fefb67c52e","Type":"ContainerDied","Data":"9b1a5d16d8e8789990a1c40930be8be8d3a33560866a0caaf05cdf5bf7149933"} Sep 17 18:09:47.486359 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:47.486339 2564 generic.go:356] "Generic (PLEG): container finished" podID="3c2262c4-3f17-4ff5-be69-b42754e3ce96" containerID="423bfb10d8f7fff60403b07c18c91a797f53a5df0a80f8b3343a057be6528c91" exitCode=0 Sep 17 18:09:47.486451 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:47.486437 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" event={"ID":"3c2262c4-3f17-4ff5-be69-b42754e3ce96","Type":"ContainerDied","Data":"423bfb10d8f7fff60403b07c18c91a797f53a5df0a80f8b3343a057be6528c91"} Sep 17 18:09:47.488408 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:47.488386 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c363b98-eadf-468d-b0ba-a60e5ad25e4b" containerID="4137d28db77e1af028b90969adc2a8e83743ab1a1584da8ff01e434cbaad7953" exitCode=0 Sep 17 18:09:47.488503 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:47.488415 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" event={"ID":"7c363b98-eadf-468d-b0ba-a60e5ad25e4b","Type":"ContainerDied","Data":"4137d28db77e1af028b90969adc2a8e83743ab1a1584da8ff01e434cbaad7953"} Sep 17 18:09:48.496108 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.496072 2564 generic.go:356] "Generic (PLEG): container finished" podID="43e36e9a-e2ac-4557-9a71-02bba2b20550" containerID="cf3a229643b67b036a062f7675e99170e7a77a6e58a6847ed60a618a830b0d85" exitCode=0 Sep 17 18:09:48.496484 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.496161 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" event={"ID":"43e36e9a-e2ac-4557-9a71-02bba2b20550","Type":"ContainerDied","Data":"cf3a229643b67b036a062f7675e99170e7a77a6e58a6847ed60a618a830b0d85"} Sep 17 18:09:48.498572 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.498548 2564 generic.go:356] "Generic (PLEG): container finished" podID="3c2262c4-3f17-4ff5-be69-b42754e3ce96" containerID="ff9427603dc45c1199293bef4417f8d70166916776a479bcb3b21b1aa6e05fc0" exitCode=0 Sep 17 18:09:48.498700 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.498623 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" event={"ID":"3c2262c4-3f17-4ff5-be69-b42754e3ce96","Type":"ContainerDied","Data":"ff9427603dc45c1199293bef4417f8d70166916776a479bcb3b21b1aa6e05fc0"} Sep 17 18:09:48.501183 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.501160 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c363b98-eadf-468d-b0ba-a60e5ad25e4b" containerID="38d1458f22a952ae0c2ee8d3508f78c503ee6b0e2802ba52e80176c5d18382cd" exitCode=0 Sep 17 18:09:48.501297 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.501241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" event={"ID":"7c363b98-eadf-468d-b0ba-a60e5ad25e4b","Type":"ContainerDied","Data":"38d1458f22a952ae0c2ee8d3508f78c503ee6b0e2802ba52e80176c5d18382cd"} Sep 17 18:09:48.640755 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.640731 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" Sep 17 18:09:48.694356 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.694325 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zj48\" (UniqueName: \"kubernetes.io/projected/2e903e93-9c1b-41be-94b1-71fefb67c52e-kube-api-access-8zj48\") pod \"2e903e93-9c1b-41be-94b1-71fefb67c52e\" (UID: \"2e903e93-9c1b-41be-94b1-71fefb67c52e\") " Sep 17 18:09:48.694502 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.694441 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e903e93-9c1b-41be-94b1-71fefb67c52e-util\") pod \"2e903e93-9c1b-41be-94b1-71fefb67c52e\" (UID: \"2e903e93-9c1b-41be-94b1-71fefb67c52e\") " Sep 17 18:09:48.694502 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.694471 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e903e93-9c1b-41be-94b1-71fefb67c52e-bundle\") pod \"2e903e93-9c1b-41be-94b1-71fefb67c52e\" (UID: \"2e903e93-9c1b-41be-94b1-71fefb67c52e\") " Sep 17 18:09:48.695017 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.694989 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e903e93-9c1b-41be-94b1-71fefb67c52e-bundle" (OuterVolumeSpecName: "bundle") pod "2e903e93-9c1b-41be-94b1-71fefb67c52e" (UID: "2e903e93-9c1b-41be-94b1-71fefb67c52e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:09:48.696552 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.696517 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e903e93-9c1b-41be-94b1-71fefb67c52e-kube-api-access-8zj48" (OuterVolumeSpecName: "kube-api-access-8zj48") pod "2e903e93-9c1b-41be-94b1-71fefb67c52e" (UID: "2e903e93-9c1b-41be-94b1-71fefb67c52e"). InnerVolumeSpecName "kube-api-access-8zj48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:09:48.699998 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.699970 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e903e93-9c1b-41be-94b1-71fefb67c52e-util" (OuterVolumeSpecName: "util") pod "2e903e93-9c1b-41be-94b1-71fefb67c52e" (UID: "2e903e93-9c1b-41be-94b1-71fefb67c52e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:09:48.795655 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.795594 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zj48\" (UniqueName: \"kubernetes.io/projected/2e903e93-9c1b-41be-94b1-71fefb67c52e-kube-api-access-8zj48\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:09:48.795655 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.795625 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e903e93-9c1b-41be-94b1-71fefb67c52e-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:09:48.795655 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:48.795634 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e903e93-9c1b-41be-94b1-71fefb67c52e-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:09:49.513154 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.513126 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" Sep 17 18:09:49.513511 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.513172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmxvwj" event={"ID":"2e903e93-9c1b-41be-94b1-71fefb67c52e","Type":"ContainerDied","Data":"70e22216ff224bf2d7906c38e83107ef37f91422347142544e1494215b24a956"} Sep 17 18:09:49.513511 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.513244 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70e22216ff224bf2d7906c38e83107ef37f91422347142544e1494215b24a956" Sep 17 18:09:49.658645 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.658619 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" Sep 17 18:09:49.687168 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.687144 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" Sep 17 18:09:49.691485 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.691459 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" Sep 17 18:09:49.701762 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.701740 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9z4sx\" (UniqueName: \"kubernetes.io/projected/3c2262c4-3f17-4ff5-be69-b42754e3ce96-kube-api-access-9z4sx\") pod \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\" (UID: \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\") " Sep 17 18:09:49.701867 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.701782 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c2262c4-3f17-4ff5-be69-b42754e3ce96-util\") pod \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\" (UID: \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\") " Sep 17 18:09:49.701910 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.701898 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c2262c4-3f17-4ff5-be69-b42754e3ce96-bundle\") pod \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\" (UID: \"3c2262c4-3f17-4ff5-be69-b42754e3ce96\") " Sep 17 18:09:49.702674 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.702644 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2262c4-3f17-4ff5-be69-b42754e3ce96-bundle" (OuterVolumeSpecName: "bundle") pod "3c2262c4-3f17-4ff5-be69-b42754e3ce96" (UID: "3c2262c4-3f17-4ff5-be69-b42754e3ce96"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:09:49.704433 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.704408 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2262c4-3f17-4ff5-be69-b42754e3ce96-kube-api-access-9z4sx" (OuterVolumeSpecName: "kube-api-access-9z4sx") pod "3c2262c4-3f17-4ff5-be69-b42754e3ce96" (UID: "3c2262c4-3f17-4ff5-be69-b42754e3ce96"). InnerVolumeSpecName "kube-api-access-9z4sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:09:49.707403 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.707370 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2262c4-3f17-4ff5-be69-b42754e3ce96-util" (OuterVolumeSpecName: "util") pod "3c2262c4-3f17-4ff5-be69-b42754e3ce96" (UID: "3c2262c4-3f17-4ff5-be69-b42754e3ce96"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:09:49.803120 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.803053 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-bundle\") pod \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\" (UID: \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\") " Sep 17 18:09:49.803120 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.803085 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43e36e9a-e2ac-4557-9a71-02bba2b20550-util\") pod \"43e36e9a-e2ac-4557-9a71-02bba2b20550\" (UID: \"43e36e9a-e2ac-4557-9a71-02bba2b20550\") " Sep 17 18:09:49.803263 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.803128 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-util\") pod \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\" (UID: \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\") " Sep 17 18:09:49.803263 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.803205 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43e36e9a-e2ac-4557-9a71-02bba2b20550-bundle\") pod \"43e36e9a-e2ac-4557-9a71-02bba2b20550\" (UID: \"43e36e9a-e2ac-4557-9a71-02bba2b20550\") " Sep 17 18:09:49.803263 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.803242 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m54m7\" (UniqueName: \"kubernetes.io/projected/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-kube-api-access-m54m7\") pod \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\" (UID: \"7c363b98-eadf-468d-b0ba-a60e5ad25e4b\") " Sep 17 18:09:49.803382 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.803267 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzj2f\" (UniqueName: \"kubernetes.io/projected/43e36e9a-e2ac-4557-9a71-02bba2b20550-kube-api-access-tzj2f\") pod \"43e36e9a-e2ac-4557-9a71-02bba2b20550\" (UID: \"43e36e9a-e2ac-4557-9a71-02bba2b20550\") " Sep 17 18:09:49.803590 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.803570 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c2262c4-3f17-4ff5-be69-b42754e3ce96-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:09:49.803687 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.803594 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c2262c4-3f17-4ff5-be69-b42754e3ce96-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:09:49.803687 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.803608 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z4sx\" (UniqueName: \"kubernetes.io/projected/3c2262c4-3f17-4ff5-be69-b42754e3ce96-kube-api-access-9z4sx\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:09:49.803687 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.803633 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-bundle" (OuterVolumeSpecName: "bundle") pod "7c363b98-eadf-468d-b0ba-a60e5ad25e4b" (UID: "7c363b98-eadf-468d-b0ba-a60e5ad25e4b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:09:49.803889 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.803867 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e36e9a-e2ac-4557-9a71-02bba2b20550-bundle" (OuterVolumeSpecName: "bundle") pod "43e36e9a-e2ac-4557-9a71-02bba2b20550" (UID: "43e36e9a-e2ac-4557-9a71-02bba2b20550"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:09:49.805541 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.805501 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e36e9a-e2ac-4557-9a71-02bba2b20550-kube-api-access-tzj2f" (OuterVolumeSpecName: "kube-api-access-tzj2f") pod "43e36e9a-e2ac-4557-9a71-02bba2b20550" (UID: "43e36e9a-e2ac-4557-9a71-02bba2b20550"). InnerVolumeSpecName "kube-api-access-tzj2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:09:49.805884 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.805865 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-kube-api-access-m54m7" (OuterVolumeSpecName: "kube-api-access-m54m7") pod "7c363b98-eadf-468d-b0ba-a60e5ad25e4b" (UID: "7c363b98-eadf-468d-b0ba-a60e5ad25e4b"). InnerVolumeSpecName "kube-api-access-m54m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:09:49.809555 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.809513 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-util" (OuterVolumeSpecName: "util") pod "7c363b98-eadf-468d-b0ba-a60e5ad25e4b" (UID: "7c363b98-eadf-468d-b0ba-a60e5ad25e4b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:09:49.809634 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.809620 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e36e9a-e2ac-4557-9a71-02bba2b20550-util" (OuterVolumeSpecName: "util") pod "43e36e9a-e2ac-4557-9a71-02bba2b20550" (UID: "43e36e9a-e2ac-4557-9a71-02bba2b20550"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:09:49.904797 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.904776 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43e36e9a-e2ac-4557-9a71-02bba2b20550-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:09:49.904797 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.904797 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m54m7\" (UniqueName: \"kubernetes.io/projected/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-kube-api-access-m54m7\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:09:49.904916 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.904808 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzj2f\" (UniqueName: \"kubernetes.io/projected/43e36e9a-e2ac-4557-9a71-02bba2b20550-kube-api-access-tzj2f\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:09:49.904916 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.904817 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:09:49.904916 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.904826 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43e36e9a-e2ac-4557-9a71-02bba2b20550-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:09:49.904916 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:49.904834 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c363b98-eadf-468d-b0ba-a60e5ad25e4b-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:09:50.521148 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:50.521119 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" Sep 17 18:09:50.521516 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:50.521145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cppwb" event={"ID":"43e36e9a-e2ac-4557-9a71-02bba2b20550","Type":"ContainerDied","Data":"a160e3ececa078df868e4773b0e8cd2f972f2e4a98310ec2f20ff97d65fe8289"} Sep 17 18:09:50.521516 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:50.521186 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a160e3ececa078df868e4773b0e8cd2f972f2e4a98310ec2f20ff97d65fe8289" Sep 17 18:09:50.523147 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:50.523129 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" Sep 17 18:09:50.523244 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:50.523161 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx6gg" event={"ID":"3c2262c4-3f17-4ff5-be69-b42754e3ce96","Type":"ContainerDied","Data":"aacae33726b3ee1ca8d9e01a340fcb1901ad6c899b4ee2e2e82eaaf1d8ba9b02"} Sep 17 18:09:50.523244 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:50.523196 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aacae33726b3ee1ca8d9e01a340fcb1901ad6c899b4ee2e2e82eaaf1d8ba9b02" Sep 17 18:09:50.525287 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:50.525262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" event={"ID":"7c363b98-eadf-468d-b0ba-a60e5ad25e4b","Type":"ContainerDied","Data":"5c1d532c195a6272a5cbf454b3ad1b82ae19b1d14a8a4dcd4442806f5bcae3b5"} Sep 17 18:09:50.525374 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:50.525298 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kcjmf" Sep 17 18:09:50.525374 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:09:50.525301 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c1d532c195a6272a5cbf454b3ad1b82ae19b1d14a8a4dcd4442806f5bcae3b5" Sep 17 18:13:09.996455 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.996413 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv"] Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.996952 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c363b98-eadf-468d-b0ba-a60e5ad25e4b" containerName="util" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.996969 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c363b98-eadf-468d-b0ba-a60e5ad25e4b" containerName="util" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.996979 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e36e9a-e2ac-4557-9a71-02bba2b20550" containerName="util" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.996984 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e36e9a-e2ac-4557-9a71-02bba2b20550" containerName="util" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.996990 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e36e9a-e2ac-4557-9a71-02bba2b20550" containerName="pull" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.996996 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e36e9a-e2ac-4557-9a71-02bba2b20550" containerName="pull" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997009 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c2262c4-3f17-4ff5-be69-b42754e3ce96" containerName="pull" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997014 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c2262c4-3f17-4ff5-be69-b42754e3ce96" containerName="pull" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997021 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e36e9a-e2ac-4557-9a71-02bba2b20550" containerName="extract" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997027 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e36e9a-e2ac-4557-9a71-02bba2b20550" containerName="extract" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997037 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c2262c4-3f17-4ff5-be69-b42754e3ce96" containerName="util" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997042 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c2262c4-3f17-4ff5-be69-b42754e3ce96" containerName="util" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997048 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e903e93-9c1b-41be-94b1-71fefb67c52e" containerName="extract" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997054 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e903e93-9c1b-41be-94b1-71fefb67c52e" containerName="extract" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997060 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c363b98-eadf-468d-b0ba-a60e5ad25e4b" containerName="pull" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997065 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c363b98-eadf-468d-b0ba-a60e5ad25e4b" containerName="pull" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997071 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e903e93-9c1b-41be-94b1-71fefb67c52e" containerName="pull" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997076 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e903e93-9c1b-41be-94b1-71fefb67c52e" containerName="pull" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997081 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c363b98-eadf-468d-b0ba-a60e5ad25e4b" containerName="extract" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997086 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c363b98-eadf-468d-b0ba-a60e5ad25e4b" containerName="extract" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997095 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e903e93-9c1b-41be-94b1-71fefb67c52e" containerName="util" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997101 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e903e93-9c1b-41be-94b1-71fefb67c52e" containerName="util" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997106 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c2262c4-3f17-4ff5-be69-b42754e3ce96" containerName="extract" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997111 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c2262c4-3f17-4ff5-be69-b42754e3ce96" containerName="extract" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997176 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e36e9a-e2ac-4557-9a71-02bba2b20550" containerName="extract" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997188 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c363b98-eadf-468d-b0ba-a60e5ad25e4b" containerName="extract" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997194 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c2262c4-3f17-4ff5-be69-b42754e3ce96" containerName="extract" Sep 17 18:13:09.998926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:09.997201 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e903e93-9c1b-41be-94b1-71fefb67c52e" containerName="extract" Sep 17 18:13:10.000179 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.000023 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.004288 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.004264 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 17 18:13:10.004737 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.004718 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k9rwt\"" Sep 17 18:13:10.005107 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.005091 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 17 18:13:10.005178 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.005161 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 17 18:13:10.018475 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.018456 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv"] Sep 17 18:13:10.079122 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.079099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d29f1d01-405e-4022-a3f5-fd6426f28706-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.079224 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.079136 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d29f1d01-405e-4022-a3f5-fd6426f28706-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.079224 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.079160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjq52\" (UniqueName: \"kubernetes.io/projected/d29f1d01-405e-4022-a3f5-fd6426f28706-kube-api-access-cjq52\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.079224 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.079212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d29f1d01-405e-4022-a3f5-fd6426f28706-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.079362 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.079302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d29f1d01-405e-4022-a3f5-fd6426f28706-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.079362 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.079336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d29f1d01-405e-4022-a3f5-fd6426f28706-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.179875 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.179850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d29f1d01-405e-4022-a3f5-fd6426f28706-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.180005 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.179880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d29f1d01-405e-4022-a3f5-fd6426f28706-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.180005 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.179899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d29f1d01-405e-4022-a3f5-fd6426f28706-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.180005 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.179917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d29f1d01-405e-4022-a3f5-fd6426f28706-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.180005 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.179944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjq52\" (UniqueName: \"kubernetes.io/projected/d29f1d01-405e-4022-a3f5-fd6426f28706-kube-api-access-cjq52\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.180005 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.179972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d29f1d01-405e-4022-a3f5-fd6426f28706-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.180325 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.180291 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d29f1d01-405e-4022-a3f5-fd6426f28706-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.180402 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.180348 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d29f1d01-405e-4022-a3f5-fd6426f28706-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.180402 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.180377 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d29f1d01-405e-4022-a3f5-fd6426f28706-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.182283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.182262 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d29f1d01-405e-4022-a3f5-fd6426f28706-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.182554 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.182521 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d29f1d01-405e-4022-a3f5-fd6426f28706-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.195755 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.195733 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjq52\" (UniqueName: \"kubernetes.io/projected/d29f1d01-405e-4022-a3f5-fd6426f28706-kube-api-access-cjq52\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv\" (UID: \"d29f1d01-405e-4022-a3f5-fd6426f28706\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.311145 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.311124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:10.589883 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.589859 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv"] Sep 17 18:13:10.592106 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:13:10.592077 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29f1d01_405e_4022_a3f5_fd6426f28706.slice/crio-bb4153e50e61f4ef777f24f5d864cd1bcfabb9669abb85ca27f10af80b1a3b14 WatchSource:0}: Error finding container bb4153e50e61f4ef777f24f5d864cd1bcfabb9669abb85ca27f10af80b1a3b14: Status 404 returned error can't find the container with id bb4153e50e61f4ef777f24f5d864cd1bcfabb9669abb85ca27f10af80b1a3b14 Sep 17 18:13:10.593760 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:10.593743 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:13:11.603125 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:11.603083 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" event={"ID":"d29f1d01-405e-4022-a3f5-fd6426f28706","Type":"ContainerStarted","Data":"bb4153e50e61f4ef777f24f5d864cd1bcfabb9669abb85ca27f10af80b1a3b14"} Sep 17 18:13:12.610028 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:12.609988 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" event={"ID":"d29f1d01-405e-4022-a3f5-fd6426f28706","Type":"ContainerStarted","Data":"1c3c0b3f1d4cacb349e5d9ef9d02338e81a95da7551148c6560a2161f7f26695"} Sep 17 18:13:12.633164 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:12.633113 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" podStartSLOduration=2.622167459 podStartE2EDuration="3.633093444s" totalImagesPullingTime="1.010925985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:12.631301113 +0000 UTC m=+617.641488249" watchObservedRunningTime="2026-09-17 18:13:12.633093444 +0000 UTC m=+617.643280583" Sep 17 18:13:20.126407 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.126318 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx"] Sep 17 18:13:20.130026 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.130002 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.132680 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.132648 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 17 18:13:20.141787 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.141767 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx"] Sep 17 18:13:20.256008 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.255986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/070c27f8-c599-4724-aae0-2e65aa5a0481-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.256111 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.256023 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/070c27f8-c599-4724-aae0-2e65aa5a0481-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.256111 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.256063 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070c27f8-c599-4724-aae0-2e65aa5a0481-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.256111 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.256083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070c27f8-c599-4724-aae0-2e65aa5a0481-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.256111 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.256100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/070c27f8-c599-4724-aae0-2e65aa5a0481-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.256269 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.256114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzdqp\" (UniqueName: \"kubernetes.io/projected/070c27f8-c599-4724-aae0-2e65aa5a0481-kube-api-access-tzdqp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.312036 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.312011 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:20.312139 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.312059 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:20.324159 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.324136 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:20.357187 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.357168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070c27f8-c599-4724-aae0-2e65aa5a0481-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.357305 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.357197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070c27f8-c599-4724-aae0-2e65aa5a0481-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.357305 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.357215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/070c27f8-c599-4724-aae0-2e65aa5a0481-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.357305 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.357231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdqp\" (UniqueName: \"kubernetes.io/projected/070c27f8-c599-4724-aae0-2e65aa5a0481-kube-api-access-tzdqp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.357305 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.357258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/070c27f8-c599-4724-aae0-2e65aa5a0481-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.357496 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.357329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/070c27f8-c599-4724-aae0-2e65aa5a0481-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.357762 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.357732 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070c27f8-c599-4724-aae0-2e65aa5a0481-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.357846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.357758 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/070c27f8-c599-4724-aae0-2e65aa5a0481-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.357846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.357785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/070c27f8-c599-4724-aae0-2e65aa5a0481-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.359630 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.359603 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/070c27f8-c599-4724-aae0-2e65aa5a0481-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.359892 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.359875 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070c27f8-c599-4724-aae0-2e65aa5a0481-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.369790 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.369772 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzdqp\" (UniqueName: \"kubernetes.io/projected/070c27f8-c599-4724-aae0-2e65aa5a0481-kube-api-access-tzdqp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6czzx\" (UID: \"070c27f8-c599-4724-aae0-2e65aa5a0481\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.441358 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.441302 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:20.660781 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.660749 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv" Sep 17 18:13:20.730838 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:20.730811 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx"] Sep 17 18:13:20.730838 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:13:20.730827 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070c27f8_c599_4724_aae0_2e65aa5a0481.slice/crio-a65f4751a424b83bced8eccb14b21c85d8277d6156dac5ae8280ac8819b2105c WatchSource:0}: Error finding container a65f4751a424b83bced8eccb14b21c85d8277d6156dac5ae8280ac8819b2105c: Status 404 returned error can't find the container with id a65f4751a424b83bced8eccb14b21c85d8277d6156dac5ae8280ac8819b2105c Sep 17 18:13:21.656032 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:21.655991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" event={"ID":"070c27f8-c599-4724-aae0-2e65aa5a0481","Type":"ContainerStarted","Data":"22763858383a8d5e8560ed3b02bbc6294c6eb2aaae2b8d45a448c700c6c59367"} Sep 17 18:13:21.656388 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:21.656053 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" event={"ID":"070c27f8-c599-4724-aae0-2e65aa5a0481","Type":"ContainerStarted","Data":"a65f4751a424b83bced8eccb14b21c85d8277d6156dac5ae8280ac8819b2105c"} Sep 17 18:13:21.705884 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:21.705839 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" podStartSLOduration=1.504786432 podStartE2EDuration="1.705824504s" totalImagesPullingTime="201.038072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:21.703015859 +0000 UTC m=+626.713202995" watchObservedRunningTime="2026-09-17 18:13:21.705824504 +0000 UTC m=+626.716011679" Sep 17 18:13:30.445227 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:30.445192 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:30.445714 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:30.445644 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:30.459456 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:30.459431 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:13:30.714829 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:13:30.714744 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6czzx" Sep 17 18:15:00.146159 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:15:00.146081 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827815-vhl6d"] Sep 17 18:15:00.149471 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:15:00.149454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827815-vhl6d" Sep 17 18:15:00.152243 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:15:00.152217 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\"" Sep 17 18:15:00.159457 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:15:00.159422 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827815-vhl6d"] Sep 17 18:15:00.160916 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:15:00.160896 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827815-vhl6d" Sep 17 18:15:00.437556 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:15:00.437516 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827815-vhl6d"] Sep 17 18:15:00.439061 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:15:00.439034 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60ed2f1_0e88_49e0_886f_9ffe6f964741.slice/crio-62fc39843551c5d19f1018c5036c6ef90d28db408d9f0d1fb9b1678c99e5d4b6 WatchSource:0}: Error finding container 62fc39843551c5d19f1018c5036c6ef90d28db408d9f0d1fb9b1678c99e5d4b6: Status 404 returned error can't find the container with id 62fc39843551c5d19f1018c5036c6ef90d28db408d9f0d1fb9b1678c99e5d4b6 Sep 17 18:15:01.188957 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:15:01.188924 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827815-vhl6d" event={"ID":"d60ed2f1-0e88-49e0-886f-9ffe6f964741","Type":"ContainerStarted","Data":"62fc39843551c5d19f1018c5036c6ef90d28db408d9f0d1fb9b1678c99e5d4b6"} Sep 17 18:15:04.215361 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:15:04.215331 2564 generic.go:356] "Generic (PLEG): container finished" podID="d60ed2f1-0e88-49e0-886f-9ffe6f964741" containerID="481ba35ffcd350d67cda1f2f8def0b6b8bcbe68ad6536e7da49970798d6748a7" exitCode=0 Sep 17 18:15:04.215770 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:15:04.215385 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827815-vhl6d" event={"ID":"d60ed2f1-0e88-49e0-886f-9ffe6f964741","Type":"ContainerDied","Data":"481ba35ffcd350d67cda1f2f8def0b6b8bcbe68ad6536e7da49970798d6748a7"} Sep 17 18:15:05.347058 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:15:05.347035 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827815-vhl6d" Sep 17 18:15:06.227700 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:15:06.227673 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827815-vhl6d" Sep 17 18:15:06.227869 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:15:06.227702 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827815-vhl6d" event={"ID":"d60ed2f1-0e88-49e0-886f-9ffe6f964741","Type":"ContainerDied","Data":"62fc39843551c5d19f1018c5036c6ef90d28db408d9f0d1fb9b1678c99e5d4b6"} Sep 17 18:15:06.227869 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:15:06.227751 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62fc39843551c5d19f1018c5036c6ef90d28db408d9f0d1fb9b1678c99e5d4b6" Sep 17 18:16:19.567907 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.567820 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz"] Sep 17 18:16:19.569253 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.569224 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d60ed2f1-0e88-49e0-886f-9ffe6f964741" containerName="cleanup" Sep 17 18:16:19.569414 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.569402 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d60ed2f1-0e88-49e0-886f-9ffe6f964741" containerName="cleanup" Sep 17 18:16:19.569736 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.569721 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d60ed2f1-0e88-49e0-886f-9ffe6f964741" containerName="cleanup" Sep 17 18:16:19.575385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.575365 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.577956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.577933 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-5d8053-openshift-default-dockercfg-qtm2r\"" Sep 17 18:16:19.585020 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.585000 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz"] Sep 17 18:16:19.632658 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.632628 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj"] Sep 17 18:16:19.636361 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.636342 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.638748 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.638732 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c31adf-openshift-default-dockercfg-mjwjp\"" Sep 17 18:16:19.648659 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.648637 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj"] Sep 17 18:16:19.706497 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.706473 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9"] Sep 17 18:16:19.709589 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.709564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-workload-certs\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.709755 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.709608 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-credential-socket\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.709755 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.709645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istiod-ca-cert\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.709755 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.709710 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-data\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.709755 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.709748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-workload-socket\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.709974 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.709773 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-podinfo\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.709974 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.709797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97xmn\" (UniqueName: \"kubernetes.io/projected/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-kube-api-access-97xmn\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.709974 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.709865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-envoy\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.709974 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.709888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-token\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.710491 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.710471 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.713210 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.713192 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b18433-openshift-default-dockercfg-zkfmk\"" Sep 17 18:16:19.727433 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.727408 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9"] Sep 17 18:16:19.810668 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.810638 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-workload-socket\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.811215 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.811188 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istiod-ca-cert\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.812329 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ef44e5-6376-4ab4-960f-d6fa1104e301-istiod-ca-cert\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.812453 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812340 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsnm9\" (UniqueName: \"kubernetes.io/projected/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-kube-api-access-lsnm9\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.812453 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-data\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.812595 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812489 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-podinfo\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.812595 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-workload-socket\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.812595 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-envoy\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.812749 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812624 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-token\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.812749 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812649 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45n7m\" (UniqueName: \"kubernetes.io/projected/71ef44e5-6376-4ab4-960f-d6fa1104e301-kube-api-access-45n7m\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.812749 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812684 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-credential-socket\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.812749 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-token\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.812749 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812739 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-workload-certs\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.812993 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-podinfo\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.812993 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-data\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.812993 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-workload-socket\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.812993 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812827 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-workload-certs\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.812993 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812878 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xmn\" (UniqueName: \"kubernetes.io/projected/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-kube-api-access-97xmn\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.812993 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-envoy\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.812993 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-data\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.812993 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812961 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-token\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.813438 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.812998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-credential-socket\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.813438 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.813021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-workload-certs\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.813438 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.813061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-envoy\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.813438 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.813084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-podinfo\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.813438 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.813131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istiod-ca-cert\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.813438 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.813166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-data\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.813438 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.813195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-credential-socket\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.813438 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.813372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-workload-socket\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.813852 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.813705 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-workload-certs\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.814235 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.814204 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-credential-socket\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.815018 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.814996 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istiod-ca-cert\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.816435 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.816270 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-podinfo\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.816585 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.816560 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-envoy\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.825195 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.825109 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97xmn\" (UniqueName: \"kubernetes.io/projected/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-kube-api-access-97xmn\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.825868 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.825735 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-token\") pod \"e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.887828 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.887797 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:19.913798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.913766 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-envoy\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.913917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.913812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-podinfo\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.913917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.913854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istiod-ca-cert\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.914033 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.913994 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-data\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.914091 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.914051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-workload-socket\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.914091 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.914082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ef44e5-6376-4ab4-960f-d6fa1104e301-istiod-ca-cert\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.914192 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.914116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsnm9\" (UniqueName: \"kubernetes.io/projected/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-kube-api-access-lsnm9\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.914192 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.914158 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-envoy\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.914293 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.914210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45n7m\" (UniqueName: \"kubernetes.io/projected/71ef44e5-6376-4ab4-960f-d6fa1104e301-kube-api-access-45n7m\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.914293 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.914246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-credential-socket\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.914684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.914658 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-credential-socket\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.914832 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.914796 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-data\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.916083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.915429 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-workload-socket\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.916083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.915485 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ef44e5-6376-4ab4-960f-d6fa1104e301-istiod-ca-cert\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.916083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.915561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-token\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.916083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.915595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-workload-certs\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.916083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.915625 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-podinfo\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.916083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.915668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-workload-socket\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.916083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.915734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-data\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.916083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.915767 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-token\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.916083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.915800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-credential-socket\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.916083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.915823 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-workload-certs\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.916083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.915965 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-workload-certs\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.916718 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.916127 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-workload-certs\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.916718 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.915722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istiod-ca-cert\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.916718 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.916381 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-data\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.916718 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.916693 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-credential-socket\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.916918 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.916902 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-workload-socket\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.917476 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.917328 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-envoy\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.917679 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.917657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-podinfo\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.919507 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.919476 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-envoy\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.919680 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.919623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-podinfo\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.923400 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.923354 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsnm9\" (UniqueName: \"kubernetes.io/projected/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-kube-api-access-lsnm9\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.923400 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.923372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45n7m\" (UniqueName: \"kubernetes.io/projected/71ef44e5-6376-4ab4-960f-d6fa1104e301-kube-api-access-45n7m\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.924403 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.924381 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-token\") pod \"e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:19.926839 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.926807 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-token\") pod \"e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:19.949359 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:19.949231 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:20.028255 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.028226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:20.173397 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.173369 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz"] Sep 17 18:16:20.174262 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:16:20.174240 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92979558_f5c7_46c0_9eb2_3cdaf8edb19d.slice/crio-7fa7c882088a9576704cdaa28e1c96faf38eb317493b0a523fe4e27e491a3ff5 WatchSource:0}: Error finding container 7fa7c882088a9576704cdaa28e1c96faf38eb317493b0a523fe4e27e491a3ff5: Status 404 returned error can't find the container with id 7fa7c882088a9576704cdaa28e1c96faf38eb317493b0a523fe4e27e491a3ff5 Sep 17 18:16:20.176677 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.176631 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:20.176770 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.176720 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:20.176834 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.176770 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:20.237380 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.237358 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj"] Sep 17 18:16:20.238567 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:16:20.238538 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e8655d_b888_4cf2_b25a_e1b1eb9ab8de.slice/crio-07318f3fd5f4767969f2ca67a126173c664500cd5bbbd00ec7cdc907781b58bf WatchSource:0}: Error finding container 07318f3fd5f4767969f2ca67a126173c664500cd5bbbd00ec7cdc907781b58bf: Status 404 returned error can't find the container with id 07318f3fd5f4767969f2ca67a126173c664500cd5bbbd00ec7cdc907781b58bf Sep 17 18:16:20.241083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.241052 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:20.241159 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.241116 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:20.241159 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.241145 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:20.316378 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.316243 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9"] Sep 17 18:16:20.320517 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:16:20.320491 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ef44e5_6376_4ab4_960f_d6fa1104e301.slice/crio-774f402280d0a791207c9fe6fd8953e688f48c5713be566d918ef197f9c091cb WatchSource:0}: Error finding container 774f402280d0a791207c9fe6fd8953e688f48c5713be566d918ef197f9c091cb: Status 404 returned error can't find the container with id 774f402280d0a791207c9fe6fd8953e688f48c5713be566d918ef197f9c091cb Sep 17 18:16:20.322894 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.322858 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:20.322972 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.322939 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:20.323019 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.323004 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:20.643751 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.643574 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" event={"ID":"71ef44e5-6376-4ab4-960f-d6fa1104e301","Type":"ContainerStarted","Data":"5785402cb65d6afde4676d10790e52c7e2cb86c915ec615fdd1182e8530565ad"} Sep 17 18:16:20.643751 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.643656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" event={"ID":"71ef44e5-6376-4ab4-960f-d6fa1104e301","Type":"ContainerStarted","Data":"774f402280d0a791207c9fe6fd8953e688f48c5713be566d918ef197f9c091cb"} Sep 17 18:16:20.648005 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.647977 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" event={"ID":"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de","Type":"ContainerStarted","Data":"f7cd670d197ec8bf4796b98b55dbe1f64cafd80ab3d9aadee756be1fd05b4c7d"} Sep 17 18:16:20.648146 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.648028 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" event={"ID":"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de","Type":"ContainerStarted","Data":"07318f3fd5f4767969f2ca67a126173c664500cd5bbbd00ec7cdc907781b58bf"} Sep 17 18:16:20.650534 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.650502 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" event={"ID":"92979558-f5c7-46c0-9eb2-3cdaf8edb19d","Type":"ContainerStarted","Data":"962e5636cdce719e8589716aec68cbb9e6167d491d16ef0bfd1898fe8dd698b6"} Sep 17 18:16:20.650711 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.650586 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" event={"ID":"92979558-f5c7-46c0-9eb2-3cdaf8edb19d","Type":"ContainerStarted","Data":"7fa7c882088a9576704cdaa28e1c96faf38eb317493b0a523fe4e27e491a3ff5"} Sep 17 18:16:20.678096 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.678039 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" podStartSLOduration=1.678020218 podStartE2EDuration="1.678020218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:20.675512955 +0000 UTC m=+805.685700097" watchObservedRunningTime="2026-09-17 18:16:20.678020218 +0000 UTC m=+805.688207354" Sep 17 18:16:20.710804 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.710754 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" podStartSLOduration=1.7107354940000001 podStartE2EDuration="1.710735494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:20.709188456 +0000 UTC m=+805.719375594" watchObservedRunningTime="2026-09-17 18:16:20.710735494 +0000 UTC m=+805.720922632" Sep 17 18:16:20.736446 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.736404 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" podStartSLOduration=1.7363898880000002 podStartE2EDuration="1.736389888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:20.734255899 +0000 UTC m=+805.744443035" watchObservedRunningTime="2026-09-17 18:16:20.736389888 +0000 UTC m=+805.746577024" Sep 17 18:16:20.888284 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.888249 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:20.950378 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:20.950308 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:21.029446 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:21.029426 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:21.034264 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:21.034245 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:21.656090 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:21.656051 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:21.656991 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:21.656967 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:21.893672 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:21.893643 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:21.955032 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:21.954969 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:22.662980 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:22.662948 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:22.663379 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:22.663356 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:22.663860 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:22.663836 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:22.663969 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:22.663893 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:26.075091 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:26.075056 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8"] Sep 17 18:16:26.078819 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:26.078799 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" Sep 17 18:16:26.082154 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:26.082138 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b18433\"" Sep 17 18:16:26.090370 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:26.090348 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8"] Sep 17 18:16:26.174465 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:26.174442 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdqgn\" (UniqueName: \"kubernetes.io/projected/f521ba6e-b4b5-4372-9a32-15509e5aca8d-kube-api-access-qdqgn\") pod \"payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8\" (UID: \"f521ba6e-b4b5-4372-9a32-15509e5aca8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" Sep 17 18:16:26.174612 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:26.174493 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f521ba6e-b4b5-4372-9a32-15509e5aca8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8\" (UID: \"f521ba6e-b4b5-4372-9a32-15509e5aca8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" Sep 17 18:16:26.275507 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:26.275469 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdqgn\" (UniqueName: \"kubernetes.io/projected/f521ba6e-b4b5-4372-9a32-15509e5aca8d-kube-api-access-qdqgn\") pod \"payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8\" (UID: \"f521ba6e-b4b5-4372-9a32-15509e5aca8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" Sep 17 18:16:26.275706 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:26.275516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f521ba6e-b4b5-4372-9a32-15509e5aca8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8\" (UID: \"f521ba6e-b4b5-4372-9a32-15509e5aca8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" Sep 17 18:16:26.276253 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:26.276235 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f521ba6e-b4b5-4372-9a32-15509e5aca8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8\" (UID: \"f521ba6e-b4b5-4372-9a32-15509e5aca8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" Sep 17 18:16:26.284143 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:26.284124 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdqgn\" (UniqueName: \"kubernetes.io/projected/f521ba6e-b4b5-4372-9a32-15509e5aca8d-kube-api-access-qdqgn\") pod \"payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8\" (UID: \"f521ba6e-b4b5-4372-9a32-15509e5aca8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" Sep 17 18:16:26.390903 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:26.390826 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" Sep 17 18:16:26.673326 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:26.673296 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8"] Sep 17 18:16:26.674960 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:16:26.674932 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf521ba6e_b4b5_4372_9a32_15509e5aca8d.slice/crio-b454deef47958ad27ad51d3459c9641dd428a8cb48e2c66c6e44fec36043d442 WatchSource:0}: Error finding container b454deef47958ad27ad51d3459c9641dd428a8cb48e2c66c6e44fec36043d442: Status 404 returned error can't find the container with id b454deef47958ad27ad51d3459c9641dd428a8cb48e2c66c6e44fec36043d442 Sep 17 18:16:26.686773 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:26.686750 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" event={"ID":"f521ba6e-b4b5-4372-9a32-15509e5aca8d","Type":"ContainerStarted","Data":"b454deef47958ad27ad51d3459c9641dd428a8cb48e2c66c6e44fec36043d442"} Sep 17 18:16:27.171085 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.171049 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g"] Sep 17 18:16:27.176072 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.176053 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" Sep 17 18:16:27.179200 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.179176 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-5d8053\"" Sep 17 18:16:27.183712 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.183686 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c73ee8-66dd-4f6e-9cf9-87696e8cc69b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g\" (UID: \"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" Sep 17 18:16:27.183843 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.183724 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6b9b\" (UniqueName: \"kubernetes.io/projected/48c73ee8-66dd-4f6e-9cf9-87696e8cc69b-kube-api-access-k6b9b\") pod \"payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g\" (UID: \"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" Sep 17 18:16:27.185321 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.185296 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g"] Sep 17 18:16:27.196320 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.196293 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48"] Sep 17 18:16:27.199892 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.199872 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" Sep 17 18:16:27.202151 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.202129 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-5d8053-dockercfg-45867\"" Sep 17 18:16:27.209720 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.209699 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48"] Sep 17 18:16:27.284887 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.284858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bde362a-db00-456e-954d-263597ad8c1e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48\" (UID: \"0bde362a-db00-456e-954d-263597ad8c1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" Sep 17 18:16:27.285011 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.284921 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v9cm\" (UniqueName: \"kubernetes.io/projected/0bde362a-db00-456e-954d-263597ad8c1e-kube-api-access-7v9cm\") pod \"payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48\" (UID: \"0bde362a-db00-456e-954d-263597ad8c1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" Sep 17 18:16:27.285011 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.285005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c73ee8-66dd-4f6e-9cf9-87696e8cc69b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g\" (UID: \"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" Sep 17 18:16:27.285128 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.285033 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6b9b\" (UniqueName: \"kubernetes.io/projected/48c73ee8-66dd-4f6e-9cf9-87696e8cc69b-kube-api-access-k6b9b\") pod \"payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g\" (UID: \"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" Sep 17 18:16:27.286152 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.286131 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c73ee8-66dd-4f6e-9cf9-87696e8cc69b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g\" (UID: \"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" Sep 17 18:16:27.305031 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.304821 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6b9b\" (UniqueName: \"kubernetes.io/projected/48c73ee8-66dd-4f6e-9cf9-87696e8cc69b-kube-api-access-k6b9b\") pod \"payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g\" (UID: \"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" Sep 17 18:16:27.386009 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.385974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bde362a-db00-456e-954d-263597ad8c1e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48\" (UID: \"0bde362a-db00-456e-954d-263597ad8c1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" Sep 17 18:16:27.386185 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.386035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v9cm\" (UniqueName: \"kubernetes.io/projected/0bde362a-db00-456e-954d-263597ad8c1e-kube-api-access-7v9cm\") pod \"payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48\" (UID: \"0bde362a-db00-456e-954d-263597ad8c1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" Sep 17 18:16:27.386943 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.386921 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bde362a-db00-456e-954d-263597ad8c1e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48\" (UID: \"0bde362a-db00-456e-954d-263597ad8c1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" Sep 17 18:16:27.394978 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.394957 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v9cm\" (UniqueName: \"kubernetes.io/projected/0bde362a-db00-456e-954d-263597ad8c1e-kube-api-access-7v9cm\") pod \"payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48\" (UID: \"0bde362a-db00-456e-954d-263597ad8c1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" Sep 17 18:16:27.488205 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.488124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" Sep 17 18:16:27.511490 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.511468 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" Sep 17 18:16:27.785878 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.785846 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g"] Sep 17 18:16:27.786495 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:16:27.786463 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c73ee8_66dd_4f6e_9cf9_87696e8cc69b.slice/crio-15709a562c0dd829deb20e358b9af8623b511829f04ec009f0294bc96e090beb WatchSource:0}: Error finding container 15709a562c0dd829deb20e358b9af8623b511829f04ec009f0294bc96e090beb: Status 404 returned error can't find the container with id 15709a562c0dd829deb20e358b9af8623b511829f04ec009f0294bc96e090beb Sep 17 18:16:27.794986 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:27.794961 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48"] Sep 17 18:16:27.795424 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:16:27.795403 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bde362a_db00_456e_954d_263597ad8c1e.slice/crio-6cf9bb8b21fb622e44d1ec338573f3a17dbd9e46ee034c18385672aefd52dd24 WatchSource:0}: Error finding container 6cf9bb8b21fb622e44d1ec338573f3a17dbd9e46ee034c18385672aefd52dd24: Status 404 returned error can't find the container with id 6cf9bb8b21fb622e44d1ec338573f3a17dbd9e46ee034c18385672aefd52dd24 Sep 17 18:16:28.702415 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:28.702375 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" event={"ID":"f521ba6e-b4b5-4372-9a32-15509e5aca8d","Type":"ContainerStarted","Data":"cd25901fa6e2fc33987ab39ed4bb098dbc2827d20edaff56138f8303f0fb63ac"} Sep 17 18:16:28.702887 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:28.702451 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" Sep 17 18:16:28.704036 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:28.704006 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" event={"ID":"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b","Type":"ContainerStarted","Data":"9fe98b2d7c69ac1bbdc828b02fd8a2d89485a8ad0b1cb31e71ab9baf1587916c"} Sep 17 18:16:28.704180 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:28.704063 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" Sep 17 18:16:28.704180 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:28.704086 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" event={"ID":"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b","Type":"ContainerStarted","Data":"15709a562c0dd829deb20e358b9af8623b511829f04ec009f0294bc96e090beb"} Sep 17 18:16:28.705728 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:28.705706 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" event={"ID":"0bde362a-db00-456e-954d-263597ad8c1e","Type":"ContainerStarted","Data":"09f223bfe43074f18513c53dd9fd47629ce6b0c02e3d79e91874589396d68964"} Sep 17 18:16:28.705847 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:28.705742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" event={"ID":"0bde362a-db00-456e-954d-263597ad8c1e","Type":"ContainerStarted","Data":"6cf9bb8b21fb622e44d1ec338573f3a17dbd9e46ee034c18385672aefd52dd24"} Sep 17 18:16:28.705847 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:28.705800 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" Sep 17 18:16:28.719955 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:28.719914 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" podStartSLOduration=1.189945271 podStartE2EDuration="2.719901952s" totalImagesPullingTime="1.529956681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:28.718252315 +0000 UTC m=+813.728439450" watchObservedRunningTime="2026-09-17 18:16:28.719901952 +0000 UTC m=+813.730089087" Sep 17 18:16:28.738394 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:28.738335 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" podStartSLOduration=1.322769676 podStartE2EDuration="1.738320013s" totalImagesPullingTime="415.550337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:28.734892184 +0000 UTC m=+813.745079321" watchObservedRunningTime="2026-09-17 18:16:28.738320013 +0000 UTC m=+813.748507150" Sep 17 18:16:28.755948 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:28.755905 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" podStartSLOduration=1.3322548 podStartE2EDuration="1.755876811s" totalImagesPullingTime="423.622011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:28.75236858 +0000 UTC m=+813.762555716" watchObservedRunningTime="2026-09-17 18:16:28.755876811 +0000 UTC m=+813.766063953" Sep 17 18:16:29.870046 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:29.870015 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9"] Sep 17 18:16:29.873916 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:29.873896 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:29.876436 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:29.876410 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-015dd779-openshift-default-dockercfg-mkv4n\"" Sep 17 18:16:29.885579 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:29.885559 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9"] Sep 17 18:16:29.908860 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:29.908832 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istiod-ca-cert\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:29.908985 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:29.908890 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-data\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:29.908985 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:29.908969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-workload-certs\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:29.909133 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:29.909010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-token\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:29.909133 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:29.909094 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-workload-socket\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:29.909133 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:29.909127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-credential-socket\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:29.909285 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:29.909163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-podinfo\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:29.909285 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:29.909183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-envoy\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:29.909285 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:29.909210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sjll\" (UniqueName: \"kubernetes.io/projected/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-kube-api-access-7sjll\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.011108 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.011072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-data\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.011511 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.011487 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-data\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.011735 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.011713 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-workload-certs\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.011799 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.011777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-token\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.011853 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.011828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-workload-socket\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.011903 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.011871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-credential-socket\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.011953 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.011924 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-podinfo\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.012003 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.011960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-envoy\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.012003 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.011994 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sjll\" (UniqueName: \"kubernetes.io/projected/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-kube-api-access-7sjll\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.012097 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.012023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istiod-ca-cert\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.012980 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.012950 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istiod-ca-cert\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.013269 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.013246 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-workload-certs\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.013954 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.013927 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-workload-socket\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.014361 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.014335 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-credential-socket\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.028903 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.024949 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-envoy\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.028903 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.026812 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-podinfo\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.031489 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.031464 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-token\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.034559 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.032830 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sjll\" (UniqueName: \"kubernetes.io/projected/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-kube-api-access-7sjll\") pod \"e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.186942 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.186870 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:30.472582 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.472551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9"] Sep 17 18:16:30.473989 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:16:30.473958 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2b3744_b84f_474f_bafc_8ea0bfa5059a.slice/crio-b498a5aea3b6f0992c5384320f4340ee44ddee5f2eab2b04a6078359119d3cb1 WatchSource:0}: Error finding container b498a5aea3b6f0992c5384320f4340ee44ddee5f2eab2b04a6078359119d3cb1: Status 404 returned error can't find the container with id b498a5aea3b6f0992c5384320f4340ee44ddee5f2eab2b04a6078359119d3cb1 Sep 17 18:16:30.476761 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.476716 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:30.476865 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.476805 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:30.476865 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.476851 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:30.721292 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.721201 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" event={"ID":"7c2b3744-b84f-474f-bafc-8ea0bfa5059a","Type":"ContainerStarted","Data":"d0630d11bac31d176d7b1762b3a918098d8c6bd6a91b5769727df1a02c4bd6f4"} Sep 17 18:16:30.721441 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.721309 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" event={"ID":"7c2b3744-b84f-474f-bafc-8ea0bfa5059a","Type":"ContainerStarted","Data":"b498a5aea3b6f0992c5384320f4340ee44ddee5f2eab2b04a6078359119d3cb1"} Sep 17 18:16:30.742706 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:30.742655 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" podStartSLOduration=1.742639732 podStartE2EDuration="1.742639732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:30.740180338 +0000 UTC m=+815.750367475" watchObservedRunningTime="2026-09-17 18:16:30.742639732 +0000 UTC m=+815.752826865" Sep 17 18:16:31.188054 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:31.188020 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:31.192580 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:31.192554 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:31.727082 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:31.727054 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:31.727935 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:31.727909 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:38.009388 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.009335 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh"] Sep 17 18:16:38.031657 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.030511 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.049028 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.049002 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-99ac10-openshift-default-dockercfg-59g9p\"" Sep 17 18:16:38.076321 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.076292 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh"] Sep 17 18:16:38.154378 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.154331 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-envoy\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.154740 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.154665 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-token\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.154950 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.154916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25j4g\" (UniqueName: \"kubernetes.io/projected/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-kube-api-access-25j4g\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.155138 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.155102 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-workload-socket\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.155303 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.155282 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-credential-socket\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.155448 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.155431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istiod-ca-cert\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.155608 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.155589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-podinfo\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.155756 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.155740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-data\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.156009 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.155980 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-workload-certs\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.256851 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.256814 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25j4g\" (UniqueName: \"kubernetes.io/projected/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-kube-api-access-25j4g\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.257024 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.256884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-workload-socket\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.257024 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.256920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-credential-socket\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.257024 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.256946 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istiod-ca-cert\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.257024 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.256973 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-podinfo\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.257024 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.257016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-data\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.257318 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.257122 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-workload-certs\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.257318 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.257187 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-envoy\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.257318 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.257214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-token\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.258615 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.258587 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istiod-ca-cert\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.259369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.259344 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-workload-socket\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.259838 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.259784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-credential-socket\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.260042 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.259994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-workload-certs\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.260412 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.260371 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-data\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.268038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.268008 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-podinfo\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.273066 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.273037 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-token\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.278451 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.278425 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-envoy\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.287739 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.287710 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25j4g\" (UniqueName: \"kubernetes.io/projected/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-kube-api-access-25j4g\") pod \"e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.361822 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.361518 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:38.787610 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.787029 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh"] Sep 17 18:16:38.825930 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:16:38.825702 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4f7a4f_0dbe_424b_93ce_0215bd26e252.slice/crio-3f78d69df62f44b801bd14189dabdb8e52971da3956e600e58892dd3a4a51a8c WatchSource:0}: Error finding container 3f78d69df62f44b801bd14189dabdb8e52971da3956e600e58892dd3a4a51a8c: Status 404 returned error can't find the container with id 3f78d69df62f44b801bd14189dabdb8e52971da3956e600e58892dd3a4a51a8c Sep 17 18:16:38.828696 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.828654 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:38.828780 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.828750 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:38.828821 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:38.828793 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:39.714109 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:39.714025 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" Sep 17 18:16:39.714680 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:39.714658 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" Sep 17 18:16:39.714785 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:39.714716 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" Sep 17 18:16:39.857878 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:39.856919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" event={"ID":"1d4f7a4f-0dbe-424b-93ce-0215bd26e252","Type":"ContainerStarted","Data":"490529dc704ee80d3eb758868f2eedb4e1e45df0280253827c4ccf134276e3c1"} Sep 17 18:16:39.857878 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:39.856987 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" event={"ID":"1d4f7a4f-0dbe-424b-93ce-0215bd26e252","Type":"ContainerStarted","Data":"3f78d69df62f44b801bd14189dabdb8e52971da3956e600e58892dd3a4a51a8c"} Sep 17 18:16:39.882269 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:39.882207 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" podStartSLOduration=2.882182925 podStartE2EDuration="2.882182925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:39.88002451 +0000 UTC m=+824.890211666" watchObservedRunningTime="2026-09-17 18:16:39.882182925 +0000 UTC m=+824.892370061" Sep 17 18:16:40.362487 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:40.362448 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:40.369589 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:40.369560 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:40.866951 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:40.866915 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:40.868225 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:40.868198 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:16:44.894235 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:44.894208 2564 generic.go:356] "Generic (PLEG): container finished" podID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerID="f7cd670d197ec8bf4796b98b55dbe1f64cafd80ab3d9aadee756be1fd05b4c7d" exitCode=0 Sep 17 18:16:44.894620 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:44.894284 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" event={"ID":"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de","Type":"ContainerDied","Data":"f7cd670d197ec8bf4796b98b55dbe1f64cafd80ab3d9aadee756be1fd05b4c7d"} Sep 17 18:16:44.894751 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:44.894735 2564 scope.go:117] "RemoveContainer" containerID="f7cd670d197ec8bf4796b98b55dbe1f64cafd80ab3d9aadee756be1fd05b4c7d" Sep 17 18:16:44.896050 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:44.896022 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:44.896134 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:44.896095 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:44.896134 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:44.896127 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:45.901886 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.901849 2564 generic.go:356] "Generic (PLEG): container finished" podID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerID="5785402cb65d6afde4676d10790e52c7e2cb86c915ec615fdd1182e8530565ad" exitCode=0 Sep 17 18:16:45.902245 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.901917 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" event={"ID":"71ef44e5-6376-4ab4-960f-d6fa1104e301","Type":"ContainerDied","Data":"5785402cb65d6afde4676d10790e52c7e2cb86c915ec615fdd1182e8530565ad"} Sep 17 18:16:45.902316 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.902300 2564 scope.go:117] "RemoveContainer" containerID="5785402cb65d6afde4676d10790e52c7e2cb86c915ec615fdd1182e8530565ad" Sep 17 18:16:45.903431 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.903338 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:45.903579 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.903473 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:45.903579 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.903520 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:45.906661 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.906632 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" event={"ID":"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de","Type":"ContainerStarted","Data":"d615120c5ed6704874b8902b3a04c9d8c573c1634f63e7cc26e88ee4451a81db"} Sep 17 18:16:45.909019 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.908993 2564 generic.go:356] "Generic (PLEG): container finished" podID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerID="962e5636cdce719e8589716aec68cbb9e6167d491d16ef0bfd1898fe8dd698b6" exitCode=0 Sep 17 18:16:45.909116 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.909045 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" event={"ID":"92979558-f5c7-46c0-9eb2-3cdaf8edb19d","Type":"ContainerDied","Data":"962e5636cdce719e8589716aec68cbb9e6167d491d16ef0bfd1898fe8dd698b6"} Sep 17 18:16:45.909569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.909518 2564 scope.go:117] "RemoveContainer" containerID="962e5636cdce719e8589716aec68cbb9e6167d491d16ef0bfd1898fe8dd698b6" Sep 17 18:16:45.919100 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.919060 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:45.919211 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.919158 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:45.919275 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.919210 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:45.950688 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:45.950658 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:46.668554 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.665320 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr"] Sep 17 18:16:46.688553 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.679258 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr"] Sep 17 18:16:46.688553 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.679374 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" Sep 17 18:16:46.688553 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.683869 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-99ac10-dockercfg-qtwls\"" Sep 17 18:16:46.688553 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.684165 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-99ac10\"" Sep 17 18:16:46.810552 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.810268 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh9hj\" (UniqueName: \"kubernetes.io/projected/e9eb58fe-4a3a-4b5b-b050-583b1948236b-kube-api-access-bh9hj\") pod \"payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr\" (UID: \"e9eb58fe-4a3a-4b5b-b050-583b1948236b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" Sep 17 18:16:46.810552 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.810463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9eb58fe-4a3a-4b5b-b050-583b1948236b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr\" (UID: \"e9eb58fe-4a3a-4b5b-b050-583b1948236b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" Sep 17 18:16:46.911323 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.911294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9eb58fe-4a3a-4b5b-b050-583b1948236b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr\" (UID: \"e9eb58fe-4a3a-4b5b-b050-583b1948236b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" Sep 17 18:16:46.911848 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.911827 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh9hj\" (UniqueName: \"kubernetes.io/projected/e9eb58fe-4a3a-4b5b-b050-583b1948236b-kube-api-access-bh9hj\") pod \"payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr\" (UID: \"e9eb58fe-4a3a-4b5b-b050-583b1948236b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" Sep 17 18:16:46.912998 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.912600 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9eb58fe-4a3a-4b5b-b050-583b1948236b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr\" (UID: \"e9eb58fe-4a3a-4b5b-b050-583b1948236b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" Sep 17 18:16:46.920478 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.920416 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" event={"ID":"92979558-f5c7-46c0-9eb2-3cdaf8edb19d","Type":"ContainerStarted","Data":"ff4b1a9932bcf42b5bf3c64af2f05fb0f06690045d4cd9a5634d367b90d51161"} Sep 17 18:16:46.920847 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.920825 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh9hj\" (UniqueName: \"kubernetes.io/projected/e9eb58fe-4a3a-4b5b-b050-583b1948236b-kube-api-access-bh9hj\") pod \"payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr\" (UID: \"e9eb58fe-4a3a-4b5b-b050-583b1948236b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" Sep 17 18:16:46.923470 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.923448 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" event={"ID":"71ef44e5-6376-4ab4-960f-d6fa1104e301","Type":"ContainerStarted","Data":"401987bba49844afae4700deab854b3860c144acadc1e103827065724b71a2fe"} Sep 17 18:16:46.951108 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.951078 2564 patch_prober.go:28] interesting pod/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 17 18:16:46.951331 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:46.951305 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 17 18:16:47.016994 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.016958 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" Sep 17 18:16:47.030580 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.030064 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:47.030580 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.030186 2564 patch_prober.go:28] interesting pod/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 17 18:16:47.030580 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.030229 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 17 18:16:47.478626 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.478590 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr"] Sep 17 18:16:47.889899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.889858 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:47.890101 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.890025 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 17 18:16:47.890101 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.890071 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 17 18:16:47.934947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.934904 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" event={"ID":"e9eb58fe-4a3a-4b5b-b050-583b1948236b","Type":"ContainerStarted","Data":"f0329219d6ae19f7c14ecaed764f8cdc803d1e1311f91a99ee38bf1c5ac47994"} Sep 17 18:16:47.935428 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.934968 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" event={"ID":"e9eb58fe-4a3a-4b5b-b050-583b1948236b","Type":"ContainerStarted","Data":"665b6ec3049464b8a0c15a86ba479851dea7f4c393f6b4ecd809b6c9d26d010e"} Sep 17 18:16:47.935937 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.935906 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" Sep 17 18:16:47.954598 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.950916 2564 patch_prober.go:28] interesting pod/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 17 18:16:47.954598 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.950978 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 17 18:16:47.954598 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:47.954410 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" podStartSLOduration=1.954389614 podStartE2EDuration="1.954389614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:47.951494254 +0000 UTC m=+832.961681392" watchObservedRunningTime="2026-09-17 18:16:47.954389614 +0000 UTC m=+832.964576751" Sep 17 18:16:48.029883 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:48.029840 2564 patch_prober.go:28] interesting pod/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 17 18:16:48.030170 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:48.030132 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 17 18:16:48.889231 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:48.889187 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 17 18:16:48.889553 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:48.889483 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 17 18:16:48.950425 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:48.950390 2564 patch_prober.go:28] interesting pod/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 17 18:16:48.950994 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:48.950959 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 17 18:16:49.032119 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:49.032083 2564 patch_prober.go:28] interesting pod/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 17 18:16:49.032444 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:49.032389 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 17 18:16:49.888578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:49.888520 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 17 18:16:49.888855 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:49.888820 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 17 18:16:49.955025 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:49.954992 2564 patch_prober.go:28] interesting pod/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 17 18:16:49.955582 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:49.955553 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 17 18:16:50.029726 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:50.029624 2564 patch_prober.go:28] interesting pod/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 17 18:16:50.029994 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:50.029947 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 17 18:16:50.889131 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:50.889085 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 17 18:16:50.889304 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:50.889152 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 17 18:16:50.951179 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:50.951142 2564 patch_prober.go:28] interesting pod/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 17 18:16:50.951469 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:50.951436 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 17 18:16:51.029736 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:51.029701 2564 patch_prober.go:28] interesting pod/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 17 18:16:51.030275 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:51.030245 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 17 18:16:51.656364 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:51.656324 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:51.889248 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:51.889208 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 17 18:16:51.889459 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:51.889280 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 17 18:16:51.973627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:51.973317 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:51.975023 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:51.975003 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:51.979941 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:51.979923 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:16:52.030656 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.029358 2564 patch_prober.go:28] interesting pod/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 17 18:16:52.030656 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.029421 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 17 18:16:52.387619 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.387579 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9"] Sep 17 18:16:52.392411 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.392359 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9" Sep 17 18:16:52.396784 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.396552 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 18:16:52.396784 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.396583 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\"" Sep 17 18:16:52.411869 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.411835 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9"] Sep 17 18:16:52.505551 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.504765 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ebaa625-1bb7-4ea0-96a3-54562d2dbad5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-015dd779-sf9n9\" (UID: \"4ebaa625-1bb7-4ea0-96a3-54562d2dbad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9" Sep 17 18:16:52.605968 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.605924 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ebaa625-1bb7-4ea0-96a3-54562d2dbad5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-015dd779-sf9n9\" (UID: \"4ebaa625-1bb7-4ea0-96a3-54562d2dbad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9" Sep 17 18:16:52.607287 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.607057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ebaa625-1bb7-4ea0-96a3-54562d2dbad5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-015dd779-sf9n9\" (UID: \"4ebaa625-1bb7-4ea0-96a3-54562d2dbad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9" Sep 17 18:16:52.663114 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.663025 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:52.714316 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.714284 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9" Sep 17 18:16:52.926555 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.922688 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 17 18:16:52.926555 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:52.922749 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 17 18:16:53.030592 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:53.030269 2564 patch_prober.go:28] interesting pod/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 17 18:16:53.030592 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:53.030334 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 17 18:16:53.099728 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:53.099695 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9"] Sep 17 18:16:53.102179 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:16:53.102141 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebaa625_1bb7_4ea0_96a3_54562d2dbad5.slice/crio-e9bf7cbaf71cf517bbd96b13213c236763c2990f1fe690f6b17906d393d6d03c WatchSource:0}: Error finding container e9bf7cbaf71cf517bbd96b13213c236763c2990f1fe690f6b17906d393d6d03c: Status 404 returned error can't find the container with id e9bf7cbaf71cf517bbd96b13213c236763c2990f1fe690f6b17906d393d6d03c Sep 17 18:16:53.893900 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:53.893865 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:53.894934 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:53.894911 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:16:54.015440 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:54.015387 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9" event={"ID":"4ebaa625-1bb7-4ea0-96a3-54562d2dbad5","Type":"ContainerStarted","Data":"e7777cdeccff2cc15ef4d3589c72a9685c6c8b88c4b91a10abd14cd7eb30b4aa"} Sep 17 18:16:54.015699 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:54.015474 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9" event={"ID":"4ebaa625-1bb7-4ea0-96a3-54562d2dbad5","Type":"ContainerStarted","Data":"e9bf7cbaf71cf517bbd96b13213c236763c2990f1fe690f6b17906d393d6d03c"} Sep 17 18:16:54.028985 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:54.028954 2564 patch_prober.go:28] interesting pod/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 17 18:16:54.029200 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:54.029171 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 17 18:16:54.051553 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:54.045011 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9" podStartSLOduration=2.044977059 podStartE2EDuration="2.044977059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:54.04340382 +0000 UTC m=+839.053590957" watchObservedRunningTime="2026-09-17 18:16:54.044977059 +0000 UTC m=+839.055164196" Sep 17 18:16:55.024763 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:55.024733 2564 generic.go:356] "Generic (PLEG): container finished" podID="4ebaa625-1bb7-4ea0-96a3-54562d2dbad5" containerID="e7777cdeccff2cc15ef4d3589c72a9685c6c8b88c4b91a10abd14cd7eb30b4aa" exitCode=0 Sep 17 18:16:55.024941 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:55.024861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9" event={"ID":"4ebaa625-1bb7-4ea0-96a3-54562d2dbad5","Type":"ContainerDied","Data":"e7777cdeccff2cc15ef4d3589c72a9685c6c8b88c4b91a10abd14cd7eb30b4aa"} Sep 17 18:16:55.033027 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:55.033004 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:55.039924 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:55.039887 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:16:56.032793 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:56.032759 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerID="d0630d11bac31d176d7b1762b3a918098d8c6bd6a91b5769727df1a02c4bd6f4" exitCode=0 Sep 17 18:16:56.033223 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:56.032839 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" event={"ID":"7c2b3744-b84f-474f-bafc-8ea0bfa5059a","Type":"ContainerDied","Data":"d0630d11bac31d176d7b1762b3a918098d8c6bd6a91b5769727df1a02c4bd6f4"} Sep 17 18:16:56.034163 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:56.033562 2564 scope.go:117] "RemoveContainer" containerID="d0630d11bac31d176d7b1762b3a918098d8c6bd6a91b5769727df1a02c4bd6f4" Sep 17 18:16:56.034983 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:56.034939 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:56.035091 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:56.035029 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:56.035091 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:56.035084 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:16:56.272707 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:56.272683 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9" Sep 17 18:16:56.378047 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:56.378001 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ebaa625-1bb7-4ea0-96a3-54562d2dbad5-maas-api-service-ca\") pod \"4ebaa625-1bb7-4ea0-96a3-54562d2dbad5\" (UID: \"4ebaa625-1bb7-4ea0-96a3-54562d2dbad5\") " Sep 17 18:16:56.378351 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:56.378330 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebaa625-1bb7-4ea0-96a3-54562d2dbad5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ebaa625-1bb7-4ea0-96a3-54562d2dbad5" (UID: "4ebaa625-1bb7-4ea0-96a3-54562d2dbad5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:16:56.479753 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:56.479724 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ebaa625-1bb7-4ea0-96a3-54562d2dbad5-maas-api-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:16:57.049554 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:57.049003 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9" event={"ID":"4ebaa625-1bb7-4ea0-96a3-54562d2dbad5","Type":"ContainerDied","Data":"e9bf7cbaf71cf517bbd96b13213c236763c2990f1fe690f6b17906d393d6d03c"} Sep 17 18:16:57.049554 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:57.049066 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9bf7cbaf71cf517bbd96b13213c236763c2990f1fe690f6b17906d393d6d03c" Sep 17 18:16:57.049554 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:57.049140 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9" Sep 17 18:16:57.067572 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:57.065920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" event={"ID":"7c2b3744-b84f-474f-bafc-8ea0bfa5059a","Type":"ContainerStarted","Data":"ad8e5bb08f0ed73f757052900cb75373104b3bad4e774ee984cd7baca6894801"} Sep 17 18:16:57.187227 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:57.187162 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:16:57.188576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:57.188351 2564 patch_prober.go:28] interesting pod/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 17 18:16:57.188576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:57.188407 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 17 18:16:58.188044 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:58.188009 2564 patch_prober.go:28] interesting pod/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 17 18:16:58.188402 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:58.188078 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 17 18:16:59.188212 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:59.188172 2564 patch_prober.go:28] interesting pod/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 17 18:16:59.188718 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:59.188226 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 17 18:16:59.957735 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:16:59.957702 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" Sep 17 18:17:00.187710 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:00.187678 2564 patch_prober.go:28] interesting pod/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 17 18:17:00.187910 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:00.187743 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 17 18:17:01.188766 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:01.188277 2564 patch_prober.go:28] interesting pod/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 17 18:17:01.188766 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:01.188342 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 17 18:17:01.727863 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:01.727837 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:17:02.099327 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:02.099301 2564 generic.go:356] "Generic (PLEG): container finished" podID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerID="490529dc704ee80d3eb758868f2eedb4e1e45df0280253827c4ccf134276e3c1" exitCode=0 Sep 17 18:17:02.099485 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:02.099373 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" event={"ID":"1d4f7a4f-0dbe-424b-93ce-0215bd26e252","Type":"ContainerDied","Data":"490529dc704ee80d3eb758868f2eedb4e1e45df0280253827c4ccf134276e3c1"} Sep 17 18:17:02.099827 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:02.099813 2564 scope.go:117] "RemoveContainer" containerID="490529dc704ee80d3eb758868f2eedb4e1e45df0280253827c4ccf134276e3c1" Sep 17 18:17:02.100961 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:02.100931 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:17:02.101049 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:02.100998 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:17:02.101049 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:02.101028 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:17:02.188128 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:02.188098 2564 patch_prober.go:28] interesting pod/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 17 18:17:02.188295 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:02.188161 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 17 18:17:03.107168 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:03.107137 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" event={"ID":"1d4f7a4f-0dbe-424b-93ce-0215bd26e252","Type":"ContainerStarted","Data":"97a3fcfa913094d91c57fea652f5fe93fd5193123d0e9f33855771ffd4c23730"} Sep 17 18:17:03.192002 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:03.191970 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:17:03.192980 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:03.192958 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:17:03.363973 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:03.363873 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:17:03.364153 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:03.364002 2564 patch_prober.go:28] interesting pod/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 17 18:17:03.364153 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:03.364054 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" podUID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 17 18:17:04.362484 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:04.362445 2564 patch_prober.go:28] interesting pod/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 17 18:17:04.362864 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:04.362517 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" podUID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 17 18:17:05.216467 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:05.216432 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9"] Sep 17 18:17:05.218958 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:05.218937 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-015dd779-sf9n9"] Sep 17 18:17:05.385554 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:05.384874 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9"] Sep 17 18:17:05.385554 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:05.385123 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerName="istio-proxy" containerID="cri-o://ad8e5bb08f0ed73f757052900cb75373104b3bad4e774ee984cd7baca6894801" gracePeriod=30 Sep 17 18:17:05.391506 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:05.391476 2564 patch_prober.go:28] interesting pod/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 17 18:17:05.391965 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:05.391839 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" podUID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 17 18:17:05.468584 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:05.468475 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ebaa625-1bb7-4ea0-96a3-54562d2dbad5" path="/var/lib/kubelet/pods/4ebaa625-1bb7-4ea0-96a3-54562d2dbad5/volumes" Sep 17 18:17:06.364553 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:06.363930 2564 patch_prober.go:28] interesting pod/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 17 18:17:06.364553 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:06.363992 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" podUID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 17 18:17:07.362768 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:07.362722 2564 patch_prober.go:28] interesting pod/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 17 18:17:07.363244 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:07.362816 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" podUID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 17 18:17:08.366966 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:08.366937 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:17:08.367397 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:08.367373 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:17:08.367964 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:08.367947 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:17:10.646251 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.646232 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:17:10.741641 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.741582 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-token\") pod \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " Sep 17 18:17:10.741641 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.741616 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-envoy\") pod \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " Sep 17 18:17:10.741791 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.741647 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-podinfo\") pod \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " Sep 17 18:17:10.741791 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.741679 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-workload-socket\") pod \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " Sep 17 18:17:10.741791 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.741705 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-workload-certs\") pod \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " Sep 17 18:17:10.741791 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.741785 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-data\") pod \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " Sep 17 18:17:10.742004 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.741814 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sjll\" (UniqueName: \"kubernetes.io/projected/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-kube-api-access-7sjll\") pod \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " Sep 17 18:17:10.742004 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.741898 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istiod-ca-cert\") pod \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " Sep 17 18:17:10.742004 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.741925 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-credential-socket\") pod \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\" (UID: \"7c2b3744-b84f-474f-bafc-8ea0bfa5059a\") " Sep 17 18:17:10.742154 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.742002 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c2b3744-b84f-474f-bafc-8ea0bfa5059a" (UID: "7c2b3744-b84f-474f-bafc-8ea0bfa5059a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:17:10.742154 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.742020 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c2b3744-b84f-474f-bafc-8ea0bfa5059a" (UID: "7c2b3744-b84f-474f-bafc-8ea0bfa5059a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:17:10.742154 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.742084 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c2b3744-b84f-474f-bafc-8ea0bfa5059a" (UID: "7c2b3744-b84f-474f-bafc-8ea0bfa5059a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:17:10.742297 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.742231 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c2b3744-b84f-474f-bafc-8ea0bfa5059a" (UID: "7c2b3744-b84f-474f-bafc-8ea0bfa5059a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:17:10.742345 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.742312 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c2b3744-b84f-474f-bafc-8ea0bfa5059a" (UID: "7c2b3744-b84f-474f-bafc-8ea0bfa5059a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:17:10.742403 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.742356 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:10.742403 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.742378 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:10.742403 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.742394 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:10.742513 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.742408 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:10.744164 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.744114 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c2b3744-b84f-474f-bafc-8ea0bfa5059a" (UID: "7c2b3744-b84f-474f-bafc-8ea0bfa5059a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:17:10.744283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.744230 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-kube-api-access-7sjll" (OuterVolumeSpecName: "kube-api-access-7sjll") pod "7c2b3744-b84f-474f-bafc-8ea0bfa5059a" (UID: "7c2b3744-b84f-474f-bafc-8ea0bfa5059a"). InnerVolumeSpecName "kube-api-access-7sjll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:17:10.744348 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.744303 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c2b3744-b84f-474f-bafc-8ea0bfa5059a" (UID: "7c2b3744-b84f-474f-bafc-8ea0bfa5059a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:17:10.744386 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.744346 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c2b3744-b84f-474f-bafc-8ea0bfa5059a" (UID: "7c2b3744-b84f-474f-bafc-8ea0bfa5059a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:17:10.843703 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.843680 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sjll\" (UniqueName: \"kubernetes.io/projected/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-kube-api-access-7sjll\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:10.843703 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.843700 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:10.843829 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.843709 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:10.843829 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.843719 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:10.843829 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:10.843727 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2b3744-b84f-474f-bafc-8ea0bfa5059a-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:11.158016 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.157979 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerID="ad8e5bb08f0ed73f757052900cb75373104b3bad4e774ee984cd7baca6894801" exitCode=0 Sep 17 18:17:11.158198 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.158028 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" Sep 17 18:17:11.158198 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.158060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" event={"ID":"7c2b3744-b84f-474f-bafc-8ea0bfa5059a","Type":"ContainerDied","Data":"ad8e5bb08f0ed73f757052900cb75373104b3bad4e774ee984cd7baca6894801"} Sep 17 18:17:11.158198 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.158127 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9" event={"ID":"7c2b3744-b84f-474f-bafc-8ea0bfa5059a","Type":"ContainerDied","Data":"b498a5aea3b6f0992c5384320f4340ee44ddee5f2eab2b04a6078359119d3cb1"} Sep 17 18:17:11.158198 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.158158 2564 scope.go:117] "RemoveContainer" containerID="ad8e5bb08f0ed73f757052900cb75373104b3bad4e774ee984cd7baca6894801" Sep 17 18:17:11.169380 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.169364 2564 scope.go:117] "RemoveContainer" containerID="d0630d11bac31d176d7b1762b3a918098d8c6bd6a91b5769727df1a02c4bd6f4" Sep 17 18:17:11.178064 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.178045 2564 scope.go:117] "RemoveContainer" containerID="ad8e5bb08f0ed73f757052900cb75373104b3bad4e774ee984cd7baca6894801" Sep 17 18:17:11.178338 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:17:11.178321 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad8e5bb08f0ed73f757052900cb75373104b3bad4e774ee984cd7baca6894801\": container with ID starting with ad8e5bb08f0ed73f757052900cb75373104b3bad4e774ee984cd7baca6894801 not found: ID does not exist" containerID="ad8e5bb08f0ed73f757052900cb75373104b3bad4e774ee984cd7baca6894801" Sep 17 18:17:11.178389 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.178346 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad8e5bb08f0ed73f757052900cb75373104b3bad4e774ee984cd7baca6894801"} err="failed to get container status \"ad8e5bb08f0ed73f757052900cb75373104b3bad4e774ee984cd7baca6894801\": rpc error: code = NotFound desc = could not find container \"ad8e5bb08f0ed73f757052900cb75373104b3bad4e774ee984cd7baca6894801\": container with ID starting with ad8e5bb08f0ed73f757052900cb75373104b3bad4e774ee984cd7baca6894801 not found: ID does not exist" Sep 17 18:17:11.178389 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.178361 2564 scope.go:117] "RemoveContainer" containerID="d0630d11bac31d176d7b1762b3a918098d8c6bd6a91b5769727df1a02c4bd6f4" Sep 17 18:17:11.178625 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:17:11.178596 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0630d11bac31d176d7b1762b3a918098d8c6bd6a91b5769727df1a02c4bd6f4\": container with ID starting with d0630d11bac31d176d7b1762b3a918098d8c6bd6a91b5769727df1a02c4bd6f4 not found: ID does not exist" containerID="d0630d11bac31d176d7b1762b3a918098d8c6bd6a91b5769727df1a02c4bd6f4" Sep 17 18:17:11.178751 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.178628 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0630d11bac31d176d7b1762b3a918098d8c6bd6a91b5769727df1a02c4bd6f4"} err="failed to get container status \"d0630d11bac31d176d7b1762b3a918098d8c6bd6a91b5769727df1a02c4bd6f4\": rpc error: code = NotFound desc = could not find container \"d0630d11bac31d176d7b1762b3a918098d8c6bd6a91b5769727df1a02c4bd6f4\": container with ID starting with d0630d11bac31d176d7b1762b3a918098d8c6bd6a91b5769727df1a02c4bd6f4 not found: ID does not exist" Sep 17 18:17:11.183594 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.183558 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9"] Sep 17 18:17:11.186629 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.186610 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-015dd779-openshift-default-67789f49cd-gzkj9"] Sep 17 18:17:11.451087 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.451015 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" path="/var/lib/kubelet/pods/7c2b3744-b84f-474f-bafc-8ea0bfa5059a/volumes" Sep 17 18:17:11.682570 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.682536 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g"] Sep 17 18:17:11.682969 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.682956 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ebaa625-1bb7-4ea0-96a3-54562d2dbad5" containerName="revoke-keys" Sep 17 18:17:11.682969 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.682970 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ebaa625-1bb7-4ea0-96a3-54562d2dbad5" containerName="revoke-keys" Sep 17 18:17:11.683047 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.682993 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerName="istio-proxy" Sep 17 18:17:11.683047 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.682998 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerName="istio-proxy" Sep 17 18:17:11.683107 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.683065 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ebaa625-1bb7-4ea0-96a3-54562d2dbad5" containerName="revoke-keys" Sep 17 18:17:11.683107 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.683074 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerName="istio-proxy" Sep 17 18:17:11.683107 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.683080 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerName="istio-proxy" Sep 17 18:17:11.685976 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.685960 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" Sep 17 18:17:11.688408 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.688388 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4390f8ca\"" Sep 17 18:17:11.688613 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.688596 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4390f8ca-dockercfg-64jl9\"" Sep 17 18:17:11.699541 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.699503 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g"] Sep 17 18:17:11.707372 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.707302 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg"] Sep 17 18:17:11.707871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.707850 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerName="istio-proxy" Sep 17 18:17:11.707871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.707871 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c2b3744-b84f-474f-bafc-8ea0bfa5059a" containerName="istio-proxy" Sep 17 18:17:11.710072 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.710054 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" Sep 17 18:17:11.724295 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.724272 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg"] Sep 17 18:17:11.855321 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.855291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f198739-41d7-49bd-8346-adb9030fb6ab-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg\" (UID: \"4f198739-41d7-49bd-8346-adb9030fb6ab\") " pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" Sep 17 18:17:11.855478 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.855341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5sgp\" (UniqueName: \"kubernetes.io/projected/4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704-kube-api-access-g5sgp\") pod \"payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g\" (UID: \"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" Sep 17 18:17:11.855478 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.855393 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prkbc\" (UniqueName: \"kubernetes.io/projected/4f198739-41d7-49bd-8346-adb9030fb6ab-kube-api-access-prkbc\") pod \"payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg\" (UID: \"4f198739-41d7-49bd-8346-adb9030fb6ab\") " pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" Sep 17 18:17:11.855569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.855512 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g\" (UID: \"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" Sep 17 18:17:11.956162 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.956133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g\" (UID: \"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" Sep 17 18:17:11.956301 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.956171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f198739-41d7-49bd-8346-adb9030fb6ab-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg\" (UID: \"4f198739-41d7-49bd-8346-adb9030fb6ab\") " pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" Sep 17 18:17:11.956301 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.956198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5sgp\" (UniqueName: \"kubernetes.io/projected/4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704-kube-api-access-g5sgp\") pod \"payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g\" (UID: \"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" Sep 17 18:17:11.956301 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.956234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prkbc\" (UniqueName: \"kubernetes.io/projected/4f198739-41d7-49bd-8346-adb9030fb6ab-kube-api-access-prkbc\") pod \"payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg\" (UID: \"4f198739-41d7-49bd-8346-adb9030fb6ab\") " pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" Sep 17 18:17:11.956943 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.956926 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f198739-41d7-49bd-8346-adb9030fb6ab-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg\" (UID: \"4f198739-41d7-49bd-8346-adb9030fb6ab\") " pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" Sep 17 18:17:11.957001 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.956925 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g\" (UID: \"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" Sep 17 18:17:11.964318 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.964263 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prkbc\" (UniqueName: \"kubernetes.io/projected/4f198739-41d7-49bd-8346-adb9030fb6ab-kube-api-access-prkbc\") pod \"payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg\" (UID: \"4f198739-41d7-49bd-8346-adb9030fb6ab\") " pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" Sep 17 18:17:11.964737 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.964718 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5sgp\" (UniqueName: \"kubernetes.io/projected/4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704-kube-api-access-g5sgp\") pod \"payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g\" (UID: \"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" Sep 17 18:17:11.996357 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:11.996333 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" Sep 17 18:17:12.020807 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:12.020784 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" Sep 17 18:17:12.276324 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:12.276172 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g"] Sep 17 18:17:12.278936 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:17:12.278907 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa7df31_7faa_4c6c_bd2b_cb15bd2a9704.slice/crio-50393b95139fa825651ca7afd6ecda36411d475f28d62a6cb1a8629262458418 WatchSource:0}: Error finding container 50393b95139fa825651ca7afd6ecda36411d475f28d62a6cb1a8629262458418: Status 404 returned error can't find the container with id 50393b95139fa825651ca7afd6ecda36411d475f28d62a6cb1a8629262458418 Sep 17 18:17:12.304243 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:12.304221 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg"] Sep 17 18:17:12.315619 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:17:12.315598 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f198739_41d7_49bd_8346_adb9030fb6ab.slice/crio-cb56501a480ca30305680f3f33d3c79f7487687e46bcf4acaea5f26a14e9aba9 WatchSource:0}: Error finding container cb56501a480ca30305680f3f33d3c79f7487687e46bcf4acaea5f26a14e9aba9: Status 404 returned error can't find the container with id cb56501a480ca30305680f3f33d3c79f7487687e46bcf4acaea5f26a14e9aba9 Sep 17 18:17:13.181212 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:13.181172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" event={"ID":"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704","Type":"ContainerStarted","Data":"568abc8731672843814852d4c1aabe4192fbc6127bb1587d31f28a674c1e5f63"} Sep 17 18:17:13.181697 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:13.181664 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" Sep 17 18:17:13.181762 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:13.181695 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" event={"ID":"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704","Type":"ContainerStarted","Data":"50393b95139fa825651ca7afd6ecda36411d475f28d62a6cb1a8629262458418"} Sep 17 18:17:13.183313 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:13.183287 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" event={"ID":"4f198739-41d7-49bd-8346-adb9030fb6ab","Type":"ContainerStarted","Data":"6db2fed434f5e2930714069ddec1bcdee70796c0d377ba87045f027149866377"} Sep 17 18:17:13.183434 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:13.183329 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" event={"ID":"4f198739-41d7-49bd-8346-adb9030fb6ab","Type":"ContainerStarted","Data":"cb56501a480ca30305680f3f33d3c79f7487687e46bcf4acaea5f26a14e9aba9"} Sep 17 18:17:13.183434 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:13.183362 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" Sep 17 18:17:13.197219 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:13.197163 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" podStartSLOduration=2.197146782 podStartE2EDuration="2.197146782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:13.196566899 +0000 UTC m=+858.206754032" watchObservedRunningTime="2026-09-17 18:17:13.197146782 +0000 UTC m=+858.207333923" Sep 17 18:17:13.212249 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:13.212202 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" podStartSLOduration=2.212185336 podStartE2EDuration="2.212185336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:13.211169654 +0000 UTC m=+858.221356792" watchObservedRunningTime="2026-09-17 18:17:13.212185336 +0000 UTC m=+858.222372474" Sep 17 18:17:24.192332 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:24.192295 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" Sep 17 18:17:24.192817 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:24.192365 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" Sep 17 18:17:33.576962 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:33.576929 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt"] Sep 17 18:17:33.579716 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:33.579695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt" Sep 17 18:17:33.581913 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:33.581895 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\"" Sep 17 18:17:33.581980 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:33.581914 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 18:17:33.588796 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:33.588773 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt"] Sep 17 18:17:33.643657 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:33.643620 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1274c3-07d2-46db-a29a-88a6fe6e747e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt\" (UID: \"4e1274c3-07d2-46db-a29a-88a6fe6e747e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt" Sep 17 18:17:33.744872 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:33.744844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1274c3-07d2-46db-a29a-88a6fe6e747e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt\" (UID: \"4e1274c3-07d2-46db-a29a-88a6fe6e747e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt" Sep 17 18:17:33.745447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:33.745427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1274c3-07d2-46db-a29a-88a6fe6e747e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt\" (UID: \"4e1274c3-07d2-46db-a29a-88a6fe6e747e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt" Sep 17 18:17:33.892556 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:33.892484 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt" Sep 17 18:17:34.173054 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:34.173030 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt"] Sep 17 18:17:34.174648 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:17:34.174617 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1274c3_07d2_46db_a29a_88a6fe6e747e.slice/crio-954cdcd091cd885ec35826a3539af72a53329c817f1727769544943663a50c9c WatchSource:0}: Error finding container 954cdcd091cd885ec35826a3539af72a53329c817f1727769544943663a50c9c: Status 404 returned error can't find the container with id 954cdcd091cd885ec35826a3539af72a53329c817f1727769544943663a50c9c Sep 17 18:17:34.314927 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:34.314897 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt" event={"ID":"4e1274c3-07d2-46db-a29a-88a6fe6e747e","Type":"ContainerStarted","Data":"954cdcd091cd885ec35826a3539af72a53329c817f1727769544943663a50c9c"} Sep 17 18:17:35.321824 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:35.321792 2564 generic.go:356] "Generic (PLEG): container finished" podID="4e1274c3-07d2-46db-a29a-88a6fe6e747e" containerID="24614bbc30e806dc11d15ddda92560ffd352b7e475664d8975109e4ba471bc7a" exitCode=0 Sep 17 18:17:35.322202 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:35.321877 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt" event={"ID":"4e1274c3-07d2-46db-a29a-88a6fe6e747e","Type":"ContainerDied","Data":"24614bbc30e806dc11d15ddda92560ffd352b7e475664d8975109e4ba471bc7a"} Sep 17 18:17:36.459037 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:36.459013 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt" Sep 17 18:17:36.573142 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:36.573115 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1274c3-07d2-46db-a29a-88a6fe6e747e-maas-api-service-ca\") pod \"4e1274c3-07d2-46db-a29a-88a6fe6e747e\" (UID: \"4e1274c3-07d2-46db-a29a-88a6fe6e747e\") " Sep 17 18:17:36.573439 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:36.573416 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1274c3-07d2-46db-a29a-88a6fe6e747e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e1274c3-07d2-46db-a29a-88a6fe6e747e" (UID: "4e1274c3-07d2-46db-a29a-88a6fe6e747e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:17:36.674470 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:36.674400 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1274c3-07d2-46db-a29a-88a6fe6e747e-maas-api-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:37.336058 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:37.336031 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt" Sep 17 18:17:37.336230 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:37.336069 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt" event={"ID":"4e1274c3-07d2-46db-a29a-88a6fe6e747e","Type":"ContainerDied","Data":"954cdcd091cd885ec35826a3539af72a53329c817f1727769544943663a50c9c"} Sep 17 18:17:37.336230 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:37.336155 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="954cdcd091cd885ec35826a3539af72a53329c817f1727769544943663a50c9c" Sep 17 18:17:39.606768 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:39.606735 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg"] Sep 17 18:17:39.607222 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:39.606945 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" podUID="4f198739-41d7-49bd-8346-adb9030fb6ab" containerName="payload-processing" containerID="cri-o://6db2fed434f5e2930714069ddec1bcdee70796c0d377ba87045f027149866377" gracePeriod=30 Sep 17 18:17:39.618813 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:39.618787 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g"] Sep 17 18:17:39.619042 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:39.619020 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" podUID="4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704" containerName="payload-pre-processing" containerID="cri-o://568abc8731672843814852d4c1aabe4192fbc6127bb1587d31f28a674c1e5f63" gracePeriod=30 Sep 17 18:17:39.802193 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:39.802160 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt"] Sep 17 18:17:39.807395 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:39.807364 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4390f8ca-jd5dt"] Sep 17 18:17:39.870643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:39.870591 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" Sep 17 18:17:39.898685 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:39.898664 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" Sep 17 18:17:40.002232 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.002199 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704-plugins-config-volume\") pod \"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704\" (UID: \"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704\") " Sep 17 18:17:40.002413 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.002253 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f198739-41d7-49bd-8346-adb9030fb6ab-plugins-config-volume\") pod \"4f198739-41d7-49bd-8346-adb9030fb6ab\" (UID: \"4f198739-41d7-49bd-8346-adb9030fb6ab\") " Sep 17 18:17:40.002413 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.002322 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prkbc\" (UniqueName: \"kubernetes.io/projected/4f198739-41d7-49bd-8346-adb9030fb6ab-kube-api-access-prkbc\") pod \"4f198739-41d7-49bd-8346-adb9030fb6ab\" (UID: \"4f198739-41d7-49bd-8346-adb9030fb6ab\") " Sep 17 18:17:40.002413 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.002352 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5sgp\" (UniqueName: \"kubernetes.io/projected/4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704-kube-api-access-g5sgp\") pod \"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704\" (UID: \"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704\") " Sep 17 18:17:40.002708 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.002671 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704" (UID: "4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:17:40.002838 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.002793 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f198739-41d7-49bd-8346-adb9030fb6ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f198739-41d7-49bd-8346-adb9030fb6ab" (UID: "4f198739-41d7-49bd-8346-adb9030fb6ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:17:40.004644 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.004621 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f198739-41d7-49bd-8346-adb9030fb6ab-kube-api-access-prkbc" (OuterVolumeSpecName: "kube-api-access-prkbc") pod "4f198739-41d7-49bd-8346-adb9030fb6ab" (UID: "4f198739-41d7-49bd-8346-adb9030fb6ab"). InnerVolumeSpecName "kube-api-access-prkbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:17:40.004747 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.004731 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704-kube-api-access-g5sgp" (OuterVolumeSpecName: "kube-api-access-g5sgp") pod "4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704" (UID: "4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704"). InnerVolumeSpecName "kube-api-access-g5sgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:17:40.104148 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.104110 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5sgp\" (UniqueName: \"kubernetes.io/projected/4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704-kube-api-access-g5sgp\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:40.104148 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.104140 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:40.104148 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.104149 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f198739-41d7-49bd-8346-adb9030fb6ab-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:40.104148 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.104157 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prkbc\" (UniqueName: \"kubernetes.io/projected/4f198739-41d7-49bd-8346-adb9030fb6ab-kube-api-access-prkbc\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:17:40.361964 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.361928 2564 generic.go:356] "Generic (PLEG): container finished" podID="4f198739-41d7-49bd-8346-adb9030fb6ab" containerID="6db2fed434f5e2930714069ddec1bcdee70796c0d377ba87045f027149866377" exitCode=0 Sep 17 18:17:40.362178 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.362064 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" Sep 17 18:17:40.362178 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.362064 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" event={"ID":"4f198739-41d7-49bd-8346-adb9030fb6ab","Type":"ContainerDied","Data":"6db2fed434f5e2930714069ddec1bcdee70796c0d377ba87045f027149866377"} Sep 17 18:17:40.362178 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.362168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg" event={"ID":"4f198739-41d7-49bd-8346-adb9030fb6ab","Type":"ContainerDied","Data":"cb56501a480ca30305680f3f33d3c79f7487687e46bcf4acaea5f26a14e9aba9"} Sep 17 18:17:40.362355 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.362202 2564 scope.go:117] "RemoveContainer" containerID="6db2fed434f5e2930714069ddec1bcdee70796c0d377ba87045f027149866377" Sep 17 18:17:40.363855 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.363829 2564 generic.go:356] "Generic (PLEG): container finished" podID="4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704" containerID="568abc8731672843814852d4c1aabe4192fbc6127bb1587d31f28a674c1e5f63" exitCode=0 Sep 17 18:17:40.364012 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.363893 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" Sep 17 18:17:40.364096 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.363907 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" event={"ID":"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704","Type":"ContainerDied","Data":"568abc8731672843814852d4c1aabe4192fbc6127bb1587d31f28a674c1e5f63"} Sep 17 18:17:40.364156 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.364140 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g" event={"ID":"4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704","Type":"ContainerDied","Data":"50393b95139fa825651ca7afd6ecda36411d475f28d62a6cb1a8629262458418"} Sep 17 18:17:40.374024 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.374006 2564 scope.go:117] "RemoveContainer" containerID="6db2fed434f5e2930714069ddec1bcdee70796c0d377ba87045f027149866377" Sep 17 18:17:40.374302 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:17:40.374280 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db2fed434f5e2930714069ddec1bcdee70796c0d377ba87045f027149866377\": container with ID starting with 6db2fed434f5e2930714069ddec1bcdee70796c0d377ba87045f027149866377 not found: ID does not exist" containerID="6db2fed434f5e2930714069ddec1bcdee70796c0d377ba87045f027149866377" Sep 17 18:17:40.374386 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.374311 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db2fed434f5e2930714069ddec1bcdee70796c0d377ba87045f027149866377"} err="failed to get container status \"6db2fed434f5e2930714069ddec1bcdee70796c0d377ba87045f027149866377\": rpc error: code = NotFound desc = could not find container \"6db2fed434f5e2930714069ddec1bcdee70796c0d377ba87045f027149866377\": container with ID starting with 6db2fed434f5e2930714069ddec1bcdee70796c0d377ba87045f027149866377 not found: ID does not exist" Sep 17 18:17:40.374386 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.374328 2564 scope.go:117] "RemoveContainer" containerID="568abc8731672843814852d4c1aabe4192fbc6127bb1587d31f28a674c1e5f63" Sep 17 18:17:40.383000 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.382984 2564 scope.go:117] "RemoveContainer" containerID="568abc8731672843814852d4c1aabe4192fbc6127bb1587d31f28a674c1e5f63" Sep 17 18:17:40.383251 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:17:40.383232 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568abc8731672843814852d4c1aabe4192fbc6127bb1587d31f28a674c1e5f63\": container with ID starting with 568abc8731672843814852d4c1aabe4192fbc6127bb1587d31f28a674c1e5f63 not found: ID does not exist" containerID="568abc8731672843814852d4c1aabe4192fbc6127bb1587d31f28a674c1e5f63" Sep 17 18:17:40.383312 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.383258 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568abc8731672843814852d4c1aabe4192fbc6127bb1587d31f28a674c1e5f63"} err="failed to get container status \"568abc8731672843814852d4c1aabe4192fbc6127bb1587d31f28a674c1e5f63\": rpc error: code = NotFound desc = could not find container \"568abc8731672843814852d4c1aabe4192fbc6127bb1587d31f28a674c1e5f63\": container with ID starting with 568abc8731672843814852d4c1aabe4192fbc6127bb1587d31f28a674c1e5f63 not found: ID does not exist" Sep 17 18:17:40.390340 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.390316 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg"] Sep 17 18:17:40.392849 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.392829 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4390f8ca-75685fb6db-f6xrg"] Sep 17 18:17:40.401296 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.401276 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g"] Sep 17 18:17:40.404940 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.404922 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4390f8ca-757c8c749d-q8s2g"] Sep 17 18:17:40.650151 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.650120 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms"] Sep 17 18:17:40.650625 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.650605 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f198739-41d7-49bd-8346-adb9030fb6ab" containerName="payload-processing" Sep 17 18:17:40.650726 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.650627 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f198739-41d7-49bd-8346-adb9030fb6ab" containerName="payload-processing" Sep 17 18:17:40.650726 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.650664 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e1274c3-07d2-46db-a29a-88a6fe6e747e" containerName="revoke-keys" Sep 17 18:17:40.650726 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.650673 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e1274c3-07d2-46db-a29a-88a6fe6e747e" containerName="revoke-keys" Sep 17 18:17:40.650726 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.650693 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704" containerName="payload-pre-processing" Sep 17 18:17:40.650726 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.650701 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704" containerName="payload-pre-processing" Sep 17 18:17:40.651000 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.650797 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e1274c3-07d2-46db-a29a-88a6fe6e747e" containerName="revoke-keys" Sep 17 18:17:40.651000 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.650814 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704" containerName="payload-pre-processing" Sep 17 18:17:40.651000 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.650824 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f198739-41d7-49bd-8346-adb9030fb6ab" containerName="payload-processing" Sep 17 18:17:40.653963 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.653942 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.656466 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.656446 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7fd35bf2-openshift-default-dockercfg-5whz2\"" Sep 17 18:17:40.665451 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.665429 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms"] Sep 17 18:17:40.709597 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.709570 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-envoy\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.709719 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.709652 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istiod-ca-cert\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.709719 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.709688 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-token\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.709835 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.709723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-data\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.709835 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.709756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-workload-socket\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.709835 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.709785 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-podinfo\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.709835 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.709806 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-credential-socket\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.710036 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.709857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-workload-certs\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.710036 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.709891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2btl\" (UniqueName: \"kubernetes.io/projected/46f51656-3a6b-40bf-a04a-cbf97e9100a7-kube-api-access-w2btl\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.810970 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.810929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-token\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.811158 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.810986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-data\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.811158 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.811019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-workload-socket\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.811158 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.811047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-podinfo\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.811158 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.811072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-credential-socket\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.811158 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.811111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-workload-certs\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.811158 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.811134 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2btl\" (UniqueName: \"kubernetes.io/projected/46f51656-3a6b-40bf-a04a-cbf97e9100a7-kube-api-access-w2btl\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.811472 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.811284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-envoy\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.811472 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.811345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istiod-ca-cert\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.811472 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.811445 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-data\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.811472 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.811462 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-workload-socket\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.811662 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.811564 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-credential-socket\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.811662 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.811612 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-workload-certs\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.812019 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.811999 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istiod-ca-cert\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.813616 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.813593 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-envoy\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.813798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.813780 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-podinfo\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.818802 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.818779 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-token\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.818908 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.818800 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2btl\" (UniqueName: \"kubernetes.io/projected/46f51656-3a6b-40bf-a04a-cbf97e9100a7-kube-api-access-w2btl\") pod \"e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:40.987243 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:40.987162 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:41.266165 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:41.264571 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms"] Sep 17 18:17:41.267250 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:17:41.267224 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f51656_3a6b_40bf_a04a_cbf97e9100a7.slice/crio-da3a246729061d38d1e9ef1c9d881ac763c19a8e6fa567fe847ee0ee53dcc694 WatchSource:0}: Error finding container da3a246729061d38d1e9ef1c9d881ac763c19a8e6fa567fe847ee0ee53dcc694: Status 404 returned error can't find the container with id da3a246729061d38d1e9ef1c9d881ac763c19a8e6fa567fe847ee0ee53dcc694 Sep 17 18:17:41.269613 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:41.269578 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:17:41.269692 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:41.269646 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:17:41.269692 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:41.269673 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:17:41.374174 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:41.374138 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" event={"ID":"46f51656-3a6b-40bf-a04a-cbf97e9100a7","Type":"ContainerStarted","Data":"0380ead5d3b4c7eb2f84f8d2c0216960a0e8c6e257f5bc3fcf2d02434af83004"} Sep 17 18:17:41.374335 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:41.374205 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" event={"ID":"46f51656-3a6b-40bf-a04a-cbf97e9100a7","Type":"ContainerStarted","Data":"da3a246729061d38d1e9ef1c9d881ac763c19a8e6fa567fe847ee0ee53dcc694"} Sep 17 18:17:41.394798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:41.394750 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" podStartSLOduration=1.394734758 podStartE2EDuration="1.394734758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:41.392341777 +0000 UTC m=+886.402528913" watchObservedRunningTime="2026-09-17 18:17:41.394734758 +0000 UTC m=+886.404921895" Sep 17 18:17:41.452714 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:41.452681 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e1274c3-07d2-46db-a29a-88a6fe6e747e" path="/var/lib/kubelet/pods/4e1274c3-07d2-46db-a29a-88a6fe6e747e/volumes" Sep 17 18:17:41.453099 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:41.453080 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f198739-41d7-49bd-8346-adb9030fb6ab" path="/var/lib/kubelet/pods/4f198739-41d7-49bd-8346-adb9030fb6ab/volumes" Sep 17 18:17:41.453457 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:41.453441 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704" path="/var/lib/kubelet/pods/4fa7df31-7faa-4c6c-bd2b-cb15bd2a9704/volumes" Sep 17 18:17:41.987786 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:41.987741 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:42.992642 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:42.992613 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:43.387736 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:43.387699 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:43.388699 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:43.388673 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:17:45.814874 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:45.814842 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms"] Sep 17 18:17:45.818103 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:45.818084 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" Sep 17 18:17:45.820387 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:45.820358 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7fd35bf2-dockercfg-5xg57\"" Sep 17 18:17:45.820387 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:45.820365 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7fd35bf2\"" Sep 17 18:17:45.829747 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:45.829723 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms"] Sep 17 18:17:45.955204 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:45.955172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms\" (UID: \"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" Sep 17 18:17:45.955378 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:45.955323 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvqzz\" (UniqueName: \"kubernetes.io/projected/a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6-kube-api-access-xvqzz\") pod \"payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms\" (UID: \"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" Sep 17 18:17:46.056071 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:46.056046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms\" (UID: \"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" Sep 17 18:17:46.056225 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:46.056121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvqzz\" (UniqueName: \"kubernetes.io/projected/a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6-kube-api-access-xvqzz\") pod \"payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms\" (UID: \"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" Sep 17 18:17:46.056765 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:46.056746 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms\" (UID: \"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" Sep 17 18:17:46.064347 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:46.064328 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvqzz\" (UniqueName: \"kubernetes.io/projected/a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6-kube-api-access-xvqzz\") pod \"payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms\" (UID: \"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" Sep 17 18:17:46.129573 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:46.129465 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" Sep 17 18:17:46.402699 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:46.402661 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms"] Sep 17 18:17:46.403542 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:17:46.403484 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bdc5bf_dc88_45a4_91a8_ac7fe18274a6.slice/crio-8529891e88e743a61194b86d35d518832b332116a87beb0403d8c23ee4ec18b3 WatchSource:0}: Error finding container 8529891e88e743a61194b86d35d518832b332116a87beb0403d8c23ee4ec18b3: Status 404 returned error can't find the container with id 8529891e88e743a61194b86d35d518832b332116a87beb0403d8c23ee4ec18b3 Sep 17 18:17:47.415603 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:47.415514 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" event={"ID":"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6","Type":"ContainerStarted","Data":"be368a0a5ebd274bd4f209e3fc738df32026ad6fe1acbd45efb782b0c1d3bf2d"} Sep 17 18:17:47.415603 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:47.415584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" event={"ID":"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6","Type":"ContainerStarted","Data":"8529891e88e743a61194b86d35d518832b332116a87beb0403d8c23ee4ec18b3"} Sep 17 18:17:47.416020 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:47.415618 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" Sep 17 18:17:47.433366 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:47.433320 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" podStartSLOduration=2.43330592 podStartE2EDuration="2.43330592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:47.432071425 +0000 UTC m=+892.442258592" watchObservedRunningTime="2026-09-17 18:17:47.43330592 +0000 UTC m=+892.443493056" Sep 17 18:17:58.423543 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:17:58.423499 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" Sep 17 18:18:05.530577 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:05.530514 2564 generic.go:356] "Generic (PLEG): container finished" podID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerID="0380ead5d3b4c7eb2f84f8d2c0216960a0e8c6e257f5bc3fcf2d02434af83004" exitCode=0 Sep 17 18:18:05.531095 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:05.530590 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" event={"ID":"46f51656-3a6b-40bf-a04a-cbf97e9100a7","Type":"ContainerDied","Data":"0380ead5d3b4c7eb2f84f8d2c0216960a0e8c6e257f5bc3fcf2d02434af83004"} Sep 17 18:18:05.531095 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:05.531047 2564 scope.go:117] "RemoveContainer" containerID="0380ead5d3b4c7eb2f84f8d2c0216960a0e8c6e257f5bc3fcf2d02434af83004" Sep 17 18:18:05.532205 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:05.532178 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:18:05.532262 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:05.532246 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:18:05.532301 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:05.532280 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:18:06.542809 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:06.542754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" event={"ID":"46f51656-3a6b-40bf-a04a-cbf97e9100a7","Type":"ContainerStarted","Data":"1a0ef60d2a780d45066e2db79093c1118d707f998c48a5580116cd80ff391929"} Sep 17 18:18:06.987513 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:06.987477 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:18:06.987716 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:06.987664 2564 patch_prober.go:28] interesting pod/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 17 18:18:06.987716 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:06.987707 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" podUID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 17 18:18:07.987852 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:07.987813 2564 patch_prober.go:28] interesting pod/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 17 18:18:07.988314 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:07.987872 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" podUID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 17 18:18:08.424992 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:08.424953 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf"] Sep 17 18:18:08.427795 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:08.427779 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf" Sep 17 18:18:08.430210 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:08.430187 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 18:18:08.430315 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:08.430197 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\"" Sep 17 18:18:08.436478 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:08.436449 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf"] Sep 17 18:18:08.476423 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:08.476383 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc96232-34cf-470a-b8fb-ec1b0bcc70c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf\" (UID: \"9fc96232-34cf-470a-b8fb-ec1b0bcc70c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf" Sep 17 18:18:08.577517 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:08.577487 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc96232-34cf-470a-b8fb-ec1b0bcc70c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf\" (UID: \"9fc96232-34cf-470a-b8fb-ec1b0bcc70c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf" Sep 17 18:18:08.578123 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:08.578099 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc96232-34cf-470a-b8fb-ec1b0bcc70c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf\" (UID: \"9fc96232-34cf-470a-b8fb-ec1b0bcc70c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf" Sep 17 18:18:08.754730 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:08.754635 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf" Sep 17 18:18:08.988495 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:08.988464 2564 patch_prober.go:28] interesting pod/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 17 18:18:08.988948 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:08.988515 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" podUID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 17 18:18:09.029848 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:09.029822 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf"] Sep 17 18:18:09.030803 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:18:09.030782 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc96232_34cf_470a_b8fb_ec1b0bcc70c1.slice/crio-d401ddcac269a05be2d97463ea14d24016b75f85d7be6800fd438f5a0a80d91d WatchSource:0}: Error finding container d401ddcac269a05be2d97463ea14d24016b75f85d7be6800fd438f5a0a80d91d: Status 404 returned error can't find the container with id d401ddcac269a05be2d97463ea14d24016b75f85d7be6800fd438f5a0a80d91d Sep 17 18:18:09.568933 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:09.568898 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf" event={"ID":"9fc96232-34cf-470a-b8fb-ec1b0bcc70c1","Type":"ContainerStarted","Data":"131b271b2227ba7decdb17ee83e0f380c3a31d47b74416cb7b3190c1fc0de8fe"} Sep 17 18:18:09.569068 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:09.568951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf" event={"ID":"9fc96232-34cf-470a-b8fb-ec1b0bcc70c1","Type":"ContainerStarted","Data":"d401ddcac269a05be2d97463ea14d24016b75f85d7be6800fd438f5a0a80d91d"} Sep 17 18:18:09.586584 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:09.586521 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf" podStartSLOduration=1.586506735 podStartE2EDuration="1.586506735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:09.583054599 +0000 UTC m=+914.593241735" watchObservedRunningTime="2026-09-17 18:18:09.586506735 +0000 UTC m=+914.596693871" Sep 17 18:18:09.988090 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:09.988054 2564 patch_prober.go:28] interesting pod/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 17 18:18:09.988222 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:09.988111 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" podUID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 17 18:18:10.576597 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:10.576561 2564 generic.go:356] "Generic (PLEG): container finished" podID="9fc96232-34cf-470a-b8fb-ec1b0bcc70c1" containerID="131b271b2227ba7decdb17ee83e0f380c3a31d47b74416cb7b3190c1fc0de8fe" exitCode=0 Sep 17 18:18:10.576995 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:10.576632 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf" event={"ID":"9fc96232-34cf-470a-b8fb-ec1b0bcc70c1","Type":"ContainerDied","Data":"131b271b2227ba7decdb17ee83e0f380c3a31d47b74416cb7b3190c1fc0de8fe"} Sep 17 18:18:10.987816 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:10.987736 2564 patch_prober.go:28] interesting pod/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 17 18:18:10.987816 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:10.987791 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" podUID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 17 18:18:11.742346 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:11.742312 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf" Sep 17 18:18:11.810343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:11.810312 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc96232-34cf-470a-b8fb-ec1b0bcc70c1-maas-api-service-ca\") pod \"9fc96232-34cf-470a-b8fb-ec1b0bcc70c1\" (UID: \"9fc96232-34cf-470a-b8fb-ec1b0bcc70c1\") " Sep 17 18:18:11.810655 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:11.810635 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc96232-34cf-470a-b8fb-ec1b0bcc70c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9fc96232-34cf-470a-b8fb-ec1b0bcc70c1" (UID: "9fc96232-34cf-470a-b8fb-ec1b0bcc70c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:18:11.911904 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:11.911843 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc96232-34cf-470a-b8fb-ec1b0bcc70c1-maas-api-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:18:11.992075 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:11.992052 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:18:11.992465 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:11.992447 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:18:11.993151 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:11.993136 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:18:12.592512 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:12.592474 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf" event={"ID":"9fc96232-34cf-470a-b8fb-ec1b0bcc70c1","Type":"ContainerDied","Data":"d401ddcac269a05be2d97463ea14d24016b75f85d7be6800fd438f5a0a80d91d"} Sep 17 18:18:12.592512 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:12.592498 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf" Sep 17 18:18:12.592732 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:12.592570 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d401ddcac269a05be2d97463ea14d24016b75f85d7be6800fd438f5a0a80d91d" Sep 17 18:18:14.437501 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:14.437470 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms"] Sep 17 18:18:14.437969 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:14.437757 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" podUID="a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6" containerName="payload-pre-processing" containerID="cri-o://be368a0a5ebd274bd4f209e3fc738df32026ad6fe1acbd45efb782b0c1d3bf2d" gracePeriod=30 Sep 17 18:18:14.610880 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:14.610854 2564 generic.go:356] "Generic (PLEG): container finished" podID="a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6" containerID="be368a0a5ebd274bd4f209e3fc738df32026ad6fe1acbd45efb782b0c1d3bf2d" exitCode=0 Sep 17 18:18:14.611038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:14.610932 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" event={"ID":"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6","Type":"ContainerDied","Data":"be368a0a5ebd274bd4f209e3fc738df32026ad6fe1acbd45efb782b0c1d3bf2d"} Sep 17 18:18:14.612596 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:14.612571 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf"] Sep 17 18:18:14.617080 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:14.617053 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fd35bf2-84qbf"] Sep 17 18:18:14.716132 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:14.715794 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" Sep 17 18:18:14.841643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:14.841606 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvqzz\" (UniqueName: \"kubernetes.io/projected/a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6-kube-api-access-xvqzz\") pod \"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6\" (UID: \"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6\") " Sep 17 18:18:14.841831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:14.841713 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6-plugins-config-volume\") pod \"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6\" (UID: \"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6\") " Sep 17 18:18:14.842093 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:14.842069 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6" (UID: "a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:18:14.843971 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:14.843951 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6-kube-api-access-xvqzz" (OuterVolumeSpecName: "kube-api-access-xvqzz") pod "a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6" (UID: "a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6"). InnerVolumeSpecName "kube-api-access-xvqzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:18:14.943111 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:14.943084 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:18:14.943111 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:14.943107 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvqzz\" (UniqueName: \"kubernetes.io/projected/a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6-kube-api-access-xvqzz\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:18:15.452413 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:15.452381 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fc96232-34cf-470a-b8fb-ec1b0bcc70c1" path="/var/lib/kubelet/pods/9fc96232-34cf-470a-b8fb-ec1b0bcc70c1/volumes" Sep 17 18:18:15.618415 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:15.618388 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" Sep 17 18:18:15.618648 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:15.618421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms" event={"ID":"a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6","Type":"ContainerDied","Data":"8529891e88e743a61194b86d35d518832b332116a87beb0403d8c23ee4ec18b3"} Sep 17 18:18:15.618648 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:15.618481 2564 scope.go:117] "RemoveContainer" containerID="be368a0a5ebd274bd4f209e3fc738df32026ad6fe1acbd45efb782b0c1d3bf2d" Sep 17 18:18:15.635092 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:15.635069 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms"] Sep 17 18:18:15.641080 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:15.641060 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7fd35bf2-76b76bdb6b-m62ms"] Sep 17 18:18:15.737735 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:15.737670 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms"] Sep 17 18:18:15.737886 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:15.737863 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" podUID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerName="istio-proxy" containerID="cri-o://1a0ef60d2a780d45066e2db79093c1118d707f998c48a5580116cd80ff391929" gracePeriod=30 Sep 17 18:18:16.462199 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.462167 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk"] Sep 17 18:18:16.462938 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.462912 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6" containerName="payload-pre-processing" Sep 17 18:18:16.463045 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.462972 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6" containerName="payload-pre-processing" Sep 17 18:18:16.463045 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.463024 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc96232-34cf-470a-b8fb-ec1b0bcc70c1" containerName="revoke-keys" Sep 17 18:18:16.463045 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.463033 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc96232-34cf-470a-b8fb-ec1b0bcc70c1" containerName="revoke-keys" Sep 17 18:18:16.463197 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.463150 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc96232-34cf-470a-b8fb-ec1b0bcc70c1" containerName="revoke-keys" Sep 17 18:18:16.463197 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.463169 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6" containerName="payload-pre-processing" Sep 17 18:18:16.466165 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.466141 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.468929 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.468907 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-700d65b8-openshift-default-dockercfg-6vlfz\"" Sep 17 18:18:16.479606 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.479578 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk"] Sep 17 18:18:16.563578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.562115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-token\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.563578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.562306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-data\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.563578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.562359 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-envoy\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.563578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.562387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-credential-socket\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.563578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.562419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-workload-certs\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.563578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.562491 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-workload-socket\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.563578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.562601 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-podinfo\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.563578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.562641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd305cd-b485-40fa-a705-86165c3eadaa-istiod-ca-cert\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.563578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.562674 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rpff\" (UniqueName: \"kubernetes.io/projected/0fd305cd-b485-40fa-a705-86165c3eadaa-kube-api-access-6rpff\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.664075 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.664042 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-workload-socket\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.664239 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.664111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-podinfo\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.664239 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.664150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd305cd-b485-40fa-a705-86165c3eadaa-istiod-ca-cert\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.664239 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.664180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rpff\" (UniqueName: \"kubernetes.io/projected/0fd305cd-b485-40fa-a705-86165c3eadaa-kube-api-access-6rpff\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.664239 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.664214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-token\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.664441 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.664292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-data\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.664441 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.664332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-envoy\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.664441 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.664360 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-credential-socket\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.664441 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.664392 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-workload-certs\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.664682 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.664487 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-workload-socket\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.664781 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.664759 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-workload-certs\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.664963 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.664926 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-data\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.665062 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.664996 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-credential-socket\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.665150 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.665131 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd305cd-b485-40fa-a705-86165c3eadaa-istiod-ca-cert\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.666913 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.666889 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-podinfo\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.667023 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.666949 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-envoy\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.674011 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.673989 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rpff\" (UniqueName: \"kubernetes.io/projected/0fd305cd-b485-40fa-a705-86165c3eadaa-kube-api-access-6rpff\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.674732 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.674713 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-token\") pod \"e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:16.779327 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:16.779268 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:17.073262 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:17.073237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk"] Sep 17 18:18:17.074181 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:18:17.074156 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd305cd_b485_40fa_a705_86165c3eadaa.slice/crio-34b253782c9ff6f2a49d226041215c73786ea830adb51cec8d10d60724d8a98c WatchSource:0}: Error finding container 34b253782c9ff6f2a49d226041215c73786ea830adb51cec8d10d60724d8a98c: Status 404 returned error can't find the container with id 34b253782c9ff6f2a49d226041215c73786ea830adb51cec8d10d60724d8a98c Sep 17 18:18:17.076410 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:17.076374 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:18:17.076486 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:17.076439 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:18:17.076486 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:17.076469 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:18:17.453896 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:17.453822 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6" path="/var/lib/kubelet/pods/a0bdc5bf-dc88-45a4-91a8-ac7fe18274a6/volumes" Sep 17 18:18:17.635040 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:17.635010 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" event={"ID":"0fd305cd-b485-40fa-a705-86165c3eadaa","Type":"ContainerStarted","Data":"ecdecd51b6eecb7770854c19d6b4130d1aeee1f905a68eb7489054b0bd066af6"} Sep 17 18:18:17.635503 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:17.635070 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" event={"ID":"0fd305cd-b485-40fa-a705-86165c3eadaa","Type":"ContainerStarted","Data":"34b253782c9ff6f2a49d226041215c73786ea830adb51cec8d10d60724d8a98c"} Sep 17 18:18:17.657130 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:17.657064 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" podStartSLOduration=1.657047715 podStartE2EDuration="1.657047715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:17.654903025 +0000 UTC m=+922.665090161" watchObservedRunningTime="2026-09-17 18:18:17.657047715 +0000 UTC m=+922.667234853" Sep 17 18:18:17.780033 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:17.779965 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:18.786081 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:18.786056 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:19.647097 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:19.647061 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:19.648069 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:19.648047 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:20.998733 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:20.998680 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:18:21.111945 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.111911 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-token\") pod \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " Sep 17 18:18:21.111945 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.111953 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-data\") pod \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " Sep 17 18:18:21.112152 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.111981 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-workload-socket\") pod \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " Sep 17 18:18:21.112152 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112009 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istiod-ca-cert\") pod \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " Sep 17 18:18:21.112152 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112054 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-podinfo\") pod \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " Sep 17 18:18:21.112152 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112119 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-envoy\") pod \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " Sep 17 18:18:21.112152 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112139 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-workload-certs\") pod \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " Sep 17 18:18:21.112424 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112159 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2btl\" (UniqueName: \"kubernetes.io/projected/46f51656-3a6b-40bf-a04a-cbf97e9100a7-kube-api-access-w2btl\") pod \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " Sep 17 18:18:21.112424 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112206 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-credential-socket\") pod \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\" (UID: \"46f51656-3a6b-40bf-a04a-cbf97e9100a7\") " Sep 17 18:18:21.112424 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112233 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-data" (OuterVolumeSpecName: "istio-data") pod "46f51656-3a6b-40bf-a04a-cbf97e9100a7" (UID: "46f51656-3a6b-40bf-a04a-cbf97e9100a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:18:21.112424 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112278 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "46f51656-3a6b-40bf-a04a-cbf97e9100a7" (UID: "46f51656-3a6b-40bf-a04a-cbf97e9100a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:18:21.112424 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112362 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "46f51656-3a6b-40bf-a04a-cbf97e9100a7" (UID: "46f51656-3a6b-40bf-a04a-cbf97e9100a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:18:21.112707 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112435 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "46f51656-3a6b-40bf-a04a-cbf97e9100a7" (UID: "46f51656-3a6b-40bf-a04a-cbf97e9100a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:18:21.112707 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112505 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "46f51656-3a6b-40bf-a04a-cbf97e9100a7" (UID: "46f51656-3a6b-40bf-a04a-cbf97e9100a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:18:21.112707 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112684 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:18:21.112707 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112705 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:18:21.112866 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112718 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:18:21.112866 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112728 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:18:21.112866 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.112736 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:18:21.114714 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.114694 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-token" (OuterVolumeSpecName: "istio-token") pod "46f51656-3a6b-40bf-a04a-cbf97e9100a7" (UID: "46f51656-3a6b-40bf-a04a-cbf97e9100a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:18:21.114797 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.114776 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "46f51656-3a6b-40bf-a04a-cbf97e9100a7" (UID: "46f51656-3a6b-40bf-a04a-cbf97e9100a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:18:21.114925 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.114907 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "46f51656-3a6b-40bf-a04a-cbf97e9100a7" (UID: "46f51656-3a6b-40bf-a04a-cbf97e9100a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:18:21.114976 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.114952 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f51656-3a6b-40bf-a04a-cbf97e9100a7-kube-api-access-w2btl" (OuterVolumeSpecName: "kube-api-access-w2btl") pod "46f51656-3a6b-40bf-a04a-cbf97e9100a7" (UID: "46f51656-3a6b-40bf-a04a-cbf97e9100a7"). InnerVolumeSpecName "kube-api-access-w2btl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:18:21.213150 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.213127 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:18:21.213150 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.213147 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:18:21.213299 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.213157 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f51656-3a6b-40bf-a04a-cbf97e9100a7-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:18:21.213299 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.213168 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2btl\" (UniqueName: \"kubernetes.io/projected/46f51656-3a6b-40bf-a04a-cbf97e9100a7-kube-api-access-w2btl\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:18:21.660538 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.660507 2564 generic.go:356] "Generic (PLEG): container finished" podID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerID="1a0ef60d2a780d45066e2db79093c1118d707f998c48a5580116cd80ff391929" exitCode=0 Sep 17 18:18:21.660659 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.660568 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" event={"ID":"46f51656-3a6b-40bf-a04a-cbf97e9100a7","Type":"ContainerDied","Data":"1a0ef60d2a780d45066e2db79093c1118d707f998c48a5580116cd80ff391929"} Sep 17 18:18:21.660659 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.660591 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" Sep 17 18:18:21.660659 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.660634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms" event={"ID":"46f51656-3a6b-40bf-a04a-cbf97e9100a7","Type":"ContainerDied","Data":"da3a246729061d38d1e9ef1c9d881ac763c19a8e6fa567fe847ee0ee53dcc694"} Sep 17 18:18:21.660765 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.660664 2564 scope.go:117] "RemoveContainer" containerID="1a0ef60d2a780d45066e2db79093c1118d707f998c48a5580116cd80ff391929" Sep 17 18:18:21.670946 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.670929 2564 scope.go:117] "RemoveContainer" containerID="0380ead5d3b4c7eb2f84f8d2c0216960a0e8c6e257f5bc3fcf2d02434af83004" Sep 17 18:18:21.679429 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.679412 2564 scope.go:117] "RemoveContainer" containerID="1a0ef60d2a780d45066e2db79093c1118d707f998c48a5580116cd80ff391929" Sep 17 18:18:21.679753 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:18:21.679732 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a0ef60d2a780d45066e2db79093c1118d707f998c48a5580116cd80ff391929\": container with ID starting with 1a0ef60d2a780d45066e2db79093c1118d707f998c48a5580116cd80ff391929 not found: ID does not exist" containerID="1a0ef60d2a780d45066e2db79093c1118d707f998c48a5580116cd80ff391929" Sep 17 18:18:21.679836 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.679760 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a0ef60d2a780d45066e2db79093c1118d707f998c48a5580116cd80ff391929"} err="failed to get container status \"1a0ef60d2a780d45066e2db79093c1118d707f998c48a5580116cd80ff391929\": rpc error: code = NotFound desc = could not find container \"1a0ef60d2a780d45066e2db79093c1118d707f998c48a5580116cd80ff391929\": container with ID starting with 1a0ef60d2a780d45066e2db79093c1118d707f998c48a5580116cd80ff391929 not found: ID does not exist" Sep 17 18:18:21.679836 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.679777 2564 scope.go:117] "RemoveContainer" containerID="0380ead5d3b4c7eb2f84f8d2c0216960a0e8c6e257f5bc3fcf2d02434af83004" Sep 17 18:18:21.680095 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:18:21.680032 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0380ead5d3b4c7eb2f84f8d2c0216960a0e8c6e257f5bc3fcf2d02434af83004\": container with ID starting with 0380ead5d3b4c7eb2f84f8d2c0216960a0e8c6e257f5bc3fcf2d02434af83004 not found: ID does not exist" containerID="0380ead5d3b4c7eb2f84f8d2c0216960a0e8c6e257f5bc3fcf2d02434af83004" Sep 17 18:18:21.680095 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.680059 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0380ead5d3b4c7eb2f84f8d2c0216960a0e8c6e257f5bc3fcf2d02434af83004"} err="failed to get container status \"0380ead5d3b4c7eb2f84f8d2c0216960a0e8c6e257f5bc3fcf2d02434af83004\": rpc error: code = NotFound desc = could not find container \"0380ead5d3b4c7eb2f84f8d2c0216960a0e8c6e257f5bc3fcf2d02434af83004\": container with ID starting with 0380ead5d3b4c7eb2f84f8d2c0216960a0e8c6e257f5bc3fcf2d02434af83004 not found: ID does not exist" Sep 17 18:18:21.684210 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.684185 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms"] Sep 17 18:18:21.697166 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:21.697146 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7fd35bf2-openshift-default-57b5ffffd6-jbnms"] Sep 17 18:18:23.456178 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:23.456142 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" path="/var/lib/kubelet/pods/46f51656-3a6b-40bf-a04a-cbf97e9100a7/volumes" Sep 17 18:18:24.017561 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.017514 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d"] Sep 17 18:18:24.017946 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.017934 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerName="istio-proxy" Sep 17 18:18:24.017991 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.017948 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerName="istio-proxy" Sep 17 18:18:24.018030 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.018021 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerName="istio-proxy" Sep 17 18:18:24.018060 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.018030 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerName="istio-proxy" Sep 17 18:18:24.020756 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.020741 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" Sep 17 18:18:24.023414 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.023393 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-700d65b8-dockercfg-x5nbm\"" Sep 17 18:18:24.023720 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.023706 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-700d65b8\"" Sep 17 18:18:24.032155 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.032131 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d"] Sep 17 18:18:24.077714 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.077694 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv"] Sep 17 18:18:24.078083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.078071 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerName="istio-proxy" Sep 17 18:18:24.078127 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.078086 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="46f51656-3a6b-40bf-a04a-cbf97e9100a7" containerName="istio-proxy" Sep 17 18:18:24.079967 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.079953 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" Sep 17 18:18:24.091974 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.091953 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv"] Sep 17 18:18:24.140562 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.140182 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzv7\" (UniqueName: \"kubernetes.io/projected/f5c9e7ca-c86f-495e-91aa-f5334dc49d44-kube-api-access-qkzv7\") pod \"payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d\" (UID: \"f5c9e7ca-c86f-495e-91aa-f5334dc49d44\") " pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" Sep 17 18:18:24.140562 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.140266 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c9e7ca-c86f-495e-91aa-f5334dc49d44-plugins-config-volume\") pod \"payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d\" (UID: \"f5c9e7ca-c86f-495e-91aa-f5334dc49d44\") " pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" Sep 17 18:18:24.140562 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.140307 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d1e4d1-2093-49ad-ad06-f159a5fbe394-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv\" (UID: \"f6d1e4d1-2093-49ad-ad06-f159a5fbe394\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" Sep 17 18:18:24.140562 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.140430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2fsg\" (UniqueName: \"kubernetes.io/projected/f6d1e4d1-2093-49ad-ad06-f159a5fbe394-kube-api-access-s2fsg\") pod \"payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv\" (UID: \"f6d1e4d1-2093-49ad-ad06-f159a5fbe394\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" Sep 17 18:18:24.241309 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.241278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzv7\" (UniqueName: \"kubernetes.io/projected/f5c9e7ca-c86f-495e-91aa-f5334dc49d44-kube-api-access-qkzv7\") pod \"payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d\" (UID: \"f5c9e7ca-c86f-495e-91aa-f5334dc49d44\") " pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" Sep 17 18:18:24.241494 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.241325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c9e7ca-c86f-495e-91aa-f5334dc49d44-plugins-config-volume\") pod \"payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d\" (UID: \"f5c9e7ca-c86f-495e-91aa-f5334dc49d44\") " pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" Sep 17 18:18:24.241494 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.241348 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d1e4d1-2093-49ad-ad06-f159a5fbe394-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv\" (UID: \"f6d1e4d1-2093-49ad-ad06-f159a5fbe394\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" Sep 17 18:18:24.241494 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.241407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2fsg\" (UniqueName: \"kubernetes.io/projected/f6d1e4d1-2093-49ad-ad06-f159a5fbe394-kube-api-access-s2fsg\") pod \"payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv\" (UID: \"f6d1e4d1-2093-49ad-ad06-f159a5fbe394\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" Sep 17 18:18:24.242088 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.242063 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d1e4d1-2093-49ad-ad06-f159a5fbe394-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv\" (UID: \"f6d1e4d1-2093-49ad-ad06-f159a5fbe394\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" Sep 17 18:18:24.242176 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.242071 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c9e7ca-c86f-495e-91aa-f5334dc49d44-plugins-config-volume\") pod \"payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d\" (UID: \"f5c9e7ca-c86f-495e-91aa-f5334dc49d44\") " pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" Sep 17 18:18:24.255867 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.255843 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2fsg\" (UniqueName: \"kubernetes.io/projected/f6d1e4d1-2093-49ad-ad06-f159a5fbe394-kube-api-access-s2fsg\") pod \"payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv\" (UID: \"f6d1e4d1-2093-49ad-ad06-f159a5fbe394\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" Sep 17 18:18:24.257581 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.257560 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzv7\" (UniqueName: \"kubernetes.io/projected/f5c9e7ca-c86f-495e-91aa-f5334dc49d44-kube-api-access-qkzv7\") pod \"payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d\" (UID: \"f5c9e7ca-c86f-495e-91aa-f5334dc49d44\") " pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" Sep 17 18:18:24.332468 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.332444 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" Sep 17 18:18:24.389926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.389898 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" Sep 17 18:18:24.616601 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.607853 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d"] Sep 17 18:18:24.616601 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:18:24.613422 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c9e7ca_c86f_495e_91aa_f5334dc49d44.slice/crio-b3f7e9c45a6a1076c204ec429677c9727cc9146ce0d3459e4fd8a4f052a8791b WatchSource:0}: Error finding container b3f7e9c45a6a1076c204ec429677c9727cc9146ce0d3459e4fd8a4f052a8791b: Status 404 returned error can't find the container with id b3f7e9c45a6a1076c204ec429677c9727cc9146ce0d3459e4fd8a4f052a8791b Sep 17 18:18:24.681097 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.681072 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv"] Sep 17 18:18:24.682418 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:18:24.682393 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d1e4d1_2093_49ad_ad06_f159a5fbe394.slice/crio-ccf4e1621d8a4dfc83e6ceb00f0db2ff0fd663726c3aa6d2d0a99d68d060c961 WatchSource:0}: Error finding container ccf4e1621d8a4dfc83e6ceb00f0db2ff0fd663726c3aa6d2d0a99d68d060c961: Status 404 returned error can't find the container with id ccf4e1621d8a4dfc83e6ceb00f0db2ff0fd663726c3aa6d2d0a99d68d060c961 Sep 17 18:18:24.684317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:24.684292 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" event={"ID":"f5c9e7ca-c86f-495e-91aa-f5334dc49d44","Type":"ContainerStarted","Data":"b3f7e9c45a6a1076c204ec429677c9727cc9146ce0d3459e4fd8a4f052a8791b"} Sep 17 18:18:25.692000 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:25.691965 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" event={"ID":"f5c9e7ca-c86f-495e-91aa-f5334dc49d44","Type":"ContainerStarted","Data":"a914cb6fd71b9a761df4d07e0c09dfabceb63f1499bc31001605374602e50ad6"} Sep 17 18:18:25.692461 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:25.692027 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" Sep 17 18:18:25.693686 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:25.693662 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" event={"ID":"f6d1e4d1-2093-49ad-ad06-f159a5fbe394","Type":"ContainerStarted","Data":"a7f77053a68b084b8c96c69698ece2749b074ef9c7030d316ed872937cc87654"} Sep 17 18:18:25.693811 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:25.693701 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" event={"ID":"f6d1e4d1-2093-49ad-ad06-f159a5fbe394","Type":"ContainerStarted","Data":"ccf4e1621d8a4dfc83e6ceb00f0db2ff0fd663726c3aa6d2d0a99d68d060c961"} Sep 17 18:18:25.693811 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:25.693741 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" Sep 17 18:18:25.711312 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:25.711268 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" podStartSLOduration=2.711256008 podStartE2EDuration="2.711256008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:25.70921359 +0000 UTC m=+930.719400726" watchObservedRunningTime="2026-09-17 18:18:25.711256008 +0000 UTC m=+930.721443178" Sep 17 18:18:25.727368 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:25.727319 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" podStartSLOduration=1.727302759 podStartE2EDuration="1.727302759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:25.726140323 +0000 UTC m=+930.736327488" watchObservedRunningTime="2026-09-17 18:18:25.727302759 +0000 UTC m=+930.737489899" Sep 17 18:18:36.702613 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:36.702585 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" Sep 17 18:18:36.703045 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:36.702646 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" Sep 17 18:18:45.824710 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:45.824681 2564 generic.go:356] "Generic (PLEG): container finished" podID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerID="ecdecd51b6eecb7770854c19d6b4130d1aeee1f905a68eb7489054b0bd066af6" exitCode=0 Sep 17 18:18:45.825138 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:45.824733 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" event={"ID":"0fd305cd-b485-40fa-a705-86165c3eadaa","Type":"ContainerDied","Data":"ecdecd51b6eecb7770854c19d6b4130d1aeee1f905a68eb7489054b0bd066af6"} Sep 17 18:18:45.825258 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:45.825240 2564 scope.go:117] "RemoveContainer" containerID="ecdecd51b6eecb7770854c19d6b4130d1aeee1f905a68eb7489054b0bd066af6" Sep 17 18:18:45.826349 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:45.826320 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:18:45.826411 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:45.826390 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:18:45.826446 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:45.826421 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:18:46.832830 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:46.832795 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" event={"ID":"0fd305cd-b485-40fa-a705-86165c3eadaa","Type":"ContainerStarted","Data":"66d256666113a17636b390ceaed4d2f082dbf21eba4a96202e1976bc97dafc0d"} Sep 17 18:18:47.780151 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:47.780121 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:47.781199 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:47.781164 2564 patch_prober.go:28] interesting pod/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 17 18:18:47.781320 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:47.781236 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 17 18:18:48.780345 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:48.780310 2564 patch_prober.go:28] interesting pod/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 17 18:18:48.780756 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:48.780371 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 17 18:18:48.949610 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:48.949570 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr"] Sep 17 18:18:48.953428 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:48.953403 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr" Sep 17 18:18:48.955987 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:48.955963 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\"" Sep 17 18:18:48.956450 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:48.956264 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 18:18:48.969066 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:48.969045 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr"] Sep 17 18:18:48.974109 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:48.974084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62afc752-5ca6-44d5-b288-41e4f13e63a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr\" (UID: \"62afc752-5ca6-44d5-b288-41e4f13e63a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr" Sep 17 18:18:49.075206 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:49.075167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62afc752-5ca6-44d5-b288-41e4f13e63a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr\" (UID: \"62afc752-5ca6-44d5-b288-41e4f13e63a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr" Sep 17 18:18:49.075831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:49.075812 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62afc752-5ca6-44d5-b288-41e4f13e63a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr\" (UID: \"62afc752-5ca6-44d5-b288-41e4f13e63a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr" Sep 17 18:18:49.269669 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:49.269634 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr" Sep 17 18:18:49.548602 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:49.545792 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr"] Sep 17 18:18:49.647248 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:49.647216 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:49.780140 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:49.780108 2564 patch_prober.go:28] interesting pod/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 17 18:18:49.780259 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:49.780160 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 17 18:18:49.867818 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:49.867787 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr" event={"ID":"62afc752-5ca6-44d5-b288-41e4f13e63a5","Type":"ContainerStarted","Data":"93f49147accc4a2cac56b8955d3802aea5090e28382ab4eb1bdfacb54b07f74d"} Sep 17 18:18:50.779974 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:50.779883 2564 patch_prober.go:28] interesting pod/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 17 18:18:50.779974 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:50.779952 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 17 18:18:50.875797 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:50.875763 2564 generic.go:356] "Generic (PLEG): container finished" podID="62afc752-5ca6-44d5-b288-41e4f13e63a5" containerID="0b65e41c034a9df3c35ee66cb65e5df833a2651811e89c045918379f52d87f98" exitCode=0 Sep 17 18:18:50.876127 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:50.875811 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr" event={"ID":"62afc752-5ca6-44d5-b288-41e4f13e63a5","Type":"ContainerDied","Data":"0b65e41c034a9df3c35ee66cb65e5df833a2651811e89c045918379f52d87f98"} Sep 17 18:18:51.780146 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:51.780114 2564 patch_prober.go:28] interesting pod/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 17 18:18:51.780306 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:51.780181 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 17 18:18:52.028102 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:52.028079 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr" Sep 17 18:18:52.103220 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:52.103192 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62afc752-5ca6-44d5-b288-41e4f13e63a5-maas-api-service-ca\") pod \"62afc752-5ca6-44d5-b288-41e4f13e63a5\" (UID: \"62afc752-5ca6-44d5-b288-41e4f13e63a5\") " Sep 17 18:18:52.103510 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:52.103488 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62afc752-5ca6-44d5-b288-41e4f13e63a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62afc752-5ca6-44d5-b288-41e4f13e63a5" (UID: "62afc752-5ca6-44d5-b288-41e4f13e63a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:18:52.204223 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:52.204191 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62afc752-5ca6-44d5-b288-41e4f13e63a5-maas-api-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:18:52.784128 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:52.784097 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:52.785097 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:52.785073 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:18:52.893133 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:52.893108 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr" Sep 17 18:18:52.893299 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:52.893143 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr" event={"ID":"62afc752-5ca6-44d5-b288-41e4f13e63a5","Type":"ContainerDied","Data":"93f49147accc4a2cac56b8955d3802aea5090e28382ab4eb1bdfacb54b07f74d"} Sep 17 18:18:52.893299 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:18:52.893186 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93f49147accc4a2cac56b8955d3802aea5090e28382ab4eb1bdfacb54b07f74d" Sep 17 18:19:04.456022 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.455988 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d"] Sep 17 18:19:04.456497 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.456270 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" podUID="f5c9e7ca-c86f-495e-91aa-f5334dc49d44" containerName="payload-processing" containerID="cri-o://a914cb6fd71b9a761df4d07e0c09dfabceb63f1499bc31001605374602e50ad6" gracePeriod=30 Sep 17 18:19:04.464462 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.464433 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv"] Sep 17 18:19:04.465166 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.464673 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" podUID="f6d1e4d1-2093-49ad-ad06-f159a5fbe394" containerName="payload-pre-processing" containerID="cri-o://a7f77053a68b084b8c96c69698ece2749b074ef9c7030d316ed872937cc87654" gracePeriod=30 Sep 17 18:19:04.626444 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.626402 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr"] Sep 17 18:19:04.632392 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.632357 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-700d65b8-rv7lr"] Sep 17 18:19:04.714092 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.714024 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" Sep 17 18:19:04.749733 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.749701 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" Sep 17 18:19:04.760188 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.758510 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk"] Sep 17 18:19:04.760188 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.758835 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerName="istio-proxy" containerID="cri-o://66d256666113a17636b390ceaed4d2f082dbf21eba4a96202e1976bc97dafc0d" gracePeriod=30 Sep 17 18:19:04.812060 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.812032 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkzv7\" (UniqueName: \"kubernetes.io/projected/f5c9e7ca-c86f-495e-91aa-f5334dc49d44-kube-api-access-qkzv7\") pod \"f5c9e7ca-c86f-495e-91aa-f5334dc49d44\" (UID: \"f5c9e7ca-c86f-495e-91aa-f5334dc49d44\") " Sep 17 18:19:04.812060 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.812065 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d1e4d1-2093-49ad-ad06-f159a5fbe394-plugins-config-volume\") pod \"f6d1e4d1-2093-49ad-ad06-f159a5fbe394\" (UID: \"f6d1e4d1-2093-49ad-ad06-f159a5fbe394\") " Sep 17 18:19:04.812305 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.812175 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2fsg\" (UniqueName: \"kubernetes.io/projected/f6d1e4d1-2093-49ad-ad06-f159a5fbe394-kube-api-access-s2fsg\") pod \"f6d1e4d1-2093-49ad-ad06-f159a5fbe394\" (UID: \"f6d1e4d1-2093-49ad-ad06-f159a5fbe394\") " Sep 17 18:19:04.812305 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.812204 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c9e7ca-c86f-495e-91aa-f5334dc49d44-plugins-config-volume\") pod \"f5c9e7ca-c86f-495e-91aa-f5334dc49d44\" (UID: \"f5c9e7ca-c86f-495e-91aa-f5334dc49d44\") " Sep 17 18:19:04.812587 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.812522 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d1e4d1-2093-49ad-ad06-f159a5fbe394-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6d1e4d1-2093-49ad-ad06-f159a5fbe394" (UID: "f6d1e4d1-2093-49ad-ad06-f159a5fbe394"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:19:04.812692 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.812627 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c9e7ca-c86f-495e-91aa-f5334dc49d44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5c9e7ca-c86f-495e-91aa-f5334dc49d44" (UID: "f5c9e7ca-c86f-495e-91aa-f5334dc49d44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:19:04.814293 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.814274 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c9e7ca-c86f-495e-91aa-f5334dc49d44-kube-api-access-qkzv7" (OuterVolumeSpecName: "kube-api-access-qkzv7") pod "f5c9e7ca-c86f-495e-91aa-f5334dc49d44" (UID: "f5c9e7ca-c86f-495e-91aa-f5334dc49d44"). InnerVolumeSpecName "kube-api-access-qkzv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:19:04.814359 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.814304 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d1e4d1-2093-49ad-ad06-f159a5fbe394-kube-api-access-s2fsg" (OuterVolumeSpecName: "kube-api-access-s2fsg") pod "f6d1e4d1-2093-49ad-ad06-f159a5fbe394" (UID: "f6d1e4d1-2093-49ad-ad06-f159a5fbe394"). InnerVolumeSpecName "kube-api-access-s2fsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:19:04.913254 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.913220 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2fsg\" (UniqueName: \"kubernetes.io/projected/f6d1e4d1-2093-49ad-ad06-f159a5fbe394-kube-api-access-s2fsg\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:19:04.913254 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.913249 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c9e7ca-c86f-495e-91aa-f5334dc49d44-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:19:04.913254 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.913259 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkzv7\" (UniqueName: \"kubernetes.io/projected/f5c9e7ca-c86f-495e-91aa-f5334dc49d44-kube-api-access-qkzv7\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:19:04.913487 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.913267 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d1e4d1-2093-49ad-ad06-f159a5fbe394-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:19:04.967940 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.967859 2564 generic.go:356] "Generic (PLEG): container finished" podID="f5c9e7ca-c86f-495e-91aa-f5334dc49d44" containerID="a914cb6fd71b9a761df4d07e0c09dfabceb63f1499bc31001605374602e50ad6" exitCode=0 Sep 17 18:19:04.967940 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.967912 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" Sep 17 18:19:04.968146 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.967940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" event={"ID":"f5c9e7ca-c86f-495e-91aa-f5334dc49d44","Type":"ContainerDied","Data":"a914cb6fd71b9a761df4d07e0c09dfabceb63f1499bc31001605374602e50ad6"} Sep 17 18:19:04.968146 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.967989 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d" event={"ID":"f5c9e7ca-c86f-495e-91aa-f5334dc49d44","Type":"ContainerDied","Data":"b3f7e9c45a6a1076c204ec429677c9727cc9146ce0d3459e4fd8a4f052a8791b"} Sep 17 18:19:04.968146 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.968015 2564 scope.go:117] "RemoveContainer" containerID="a914cb6fd71b9a761df4d07e0c09dfabceb63f1499bc31001605374602e50ad6" Sep 17 18:19:04.969498 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.969474 2564 generic.go:356] "Generic (PLEG): container finished" podID="f6d1e4d1-2093-49ad-ad06-f159a5fbe394" containerID="a7f77053a68b084b8c96c69698ece2749b074ef9c7030d316ed872937cc87654" exitCode=0 Sep 17 18:19:04.969649 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.969563 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" Sep 17 18:19:04.969649 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.969571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" event={"ID":"f6d1e4d1-2093-49ad-ad06-f159a5fbe394","Type":"ContainerDied","Data":"a7f77053a68b084b8c96c69698ece2749b074ef9c7030d316ed872937cc87654"} Sep 17 18:19:04.969649 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.969624 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv" event={"ID":"f6d1e4d1-2093-49ad-ad06-f159a5fbe394","Type":"ContainerDied","Data":"ccf4e1621d8a4dfc83e6ceb00f0db2ff0fd663726c3aa6d2d0a99d68d060c961"} Sep 17 18:19:04.978868 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.978849 2564 scope.go:117] "RemoveContainer" containerID="a914cb6fd71b9a761df4d07e0c09dfabceb63f1499bc31001605374602e50ad6" Sep 17 18:19:04.979146 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:19:04.979129 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a914cb6fd71b9a761df4d07e0c09dfabceb63f1499bc31001605374602e50ad6\": container with ID starting with a914cb6fd71b9a761df4d07e0c09dfabceb63f1499bc31001605374602e50ad6 not found: ID does not exist" containerID="a914cb6fd71b9a761df4d07e0c09dfabceb63f1499bc31001605374602e50ad6" Sep 17 18:19:04.979209 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.979155 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a914cb6fd71b9a761df4d07e0c09dfabceb63f1499bc31001605374602e50ad6"} err="failed to get container status \"a914cb6fd71b9a761df4d07e0c09dfabceb63f1499bc31001605374602e50ad6\": rpc error: code = NotFound desc = could not find container \"a914cb6fd71b9a761df4d07e0c09dfabceb63f1499bc31001605374602e50ad6\": container with ID starting with a914cb6fd71b9a761df4d07e0c09dfabceb63f1499bc31001605374602e50ad6 not found: ID does not exist" Sep 17 18:19:04.979209 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.979172 2564 scope.go:117] "RemoveContainer" containerID="a7f77053a68b084b8c96c69698ece2749b074ef9c7030d316ed872937cc87654" Sep 17 18:19:04.988221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.988201 2564 scope.go:117] "RemoveContainer" containerID="a7f77053a68b084b8c96c69698ece2749b074ef9c7030d316ed872937cc87654" Sep 17 18:19:04.988515 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:19:04.988493 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f77053a68b084b8c96c69698ece2749b074ef9c7030d316ed872937cc87654\": container with ID starting with a7f77053a68b084b8c96c69698ece2749b074ef9c7030d316ed872937cc87654 not found: ID does not exist" containerID="a7f77053a68b084b8c96c69698ece2749b074ef9c7030d316ed872937cc87654" Sep 17 18:19:04.988612 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.988541 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f77053a68b084b8c96c69698ece2749b074ef9c7030d316ed872937cc87654"} err="failed to get container status \"a7f77053a68b084b8c96c69698ece2749b074ef9c7030d316ed872937cc87654\": rpc error: code = NotFound desc = could not find container \"a7f77053a68b084b8c96c69698ece2749b074ef9c7030d316ed872937cc87654\": container with ID starting with a7f77053a68b084b8c96c69698ece2749b074ef9c7030d316ed872937cc87654 not found: ID does not exist" Sep 17 18:19:04.996505 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:04.996485 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d"] Sep 17 18:19:05.001579 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:05.001559 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-700d65b8-555cd59c5c-hg98d"] Sep 17 18:19:05.012746 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:05.012727 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv"] Sep 17 18:19:05.018080 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:05.018049 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-700d65b8-78cd6865f4-jhnjv"] Sep 17 18:19:05.452603 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:05.452576 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62afc752-5ca6-44d5-b288-41e4f13e63a5" path="/var/lib/kubelet/pods/62afc752-5ca6-44d5-b288-41e4f13e63a5/volumes" Sep 17 18:19:05.452950 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:05.452937 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5c9e7ca-c86f-495e-91aa-f5334dc49d44" path="/var/lib/kubelet/pods/f5c9e7ca-c86f-495e-91aa-f5334dc49d44/volumes" Sep 17 18:19:05.453260 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:05.453249 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6d1e4d1-2093-49ad-ad06-f159a5fbe394" path="/var/lib/kubelet/pods/f6d1e4d1-2093-49ad-ad06-f159a5fbe394/volumes" Sep 17 18:19:07.787182 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:07.787139 2564 patch_prober.go:28] interesting pod/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 17 18:19:07.787582 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:07.787193 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:19:10.008178 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.008147 2564 generic.go:356] "Generic (PLEG): container finished" podID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerID="66d256666113a17636b390ceaed4d2f082dbf21eba4a96202e1976bc97dafc0d" exitCode=0 Sep 17 18:19:10.008608 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.008239 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" event={"ID":"0fd305cd-b485-40fa-a705-86165c3eadaa","Type":"ContainerDied","Data":"66d256666113a17636b390ceaed4d2f082dbf21eba4a96202e1976bc97dafc0d"} Sep 17 18:19:10.008608 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.008327 2564 scope.go:117] "RemoveContainer" containerID="ecdecd51b6eecb7770854c19d6b4130d1aeee1f905a68eb7489054b0bd066af6" Sep 17 18:19:10.035978 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.035960 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:19:10.160894 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.160874 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-token\") pod \"0fd305cd-b485-40fa-a705-86165c3eadaa\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " Sep 17 18:19:10.161017 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.160903 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-workload-certs\") pod \"0fd305cd-b485-40fa-a705-86165c3eadaa\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " Sep 17 18:19:10.161017 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.160931 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-envoy\") pod \"0fd305cd-b485-40fa-a705-86165c3eadaa\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " Sep 17 18:19:10.161017 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.160952 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-credential-socket\") pod \"0fd305cd-b485-40fa-a705-86165c3eadaa\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " Sep 17 18:19:10.161017 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.160976 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-data\") pod \"0fd305cd-b485-40fa-a705-86165c3eadaa\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " Sep 17 18:19:10.161206 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.161018 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-workload-socket\") pod \"0fd305cd-b485-40fa-a705-86165c3eadaa\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " Sep 17 18:19:10.161206 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.161076 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-podinfo\") pod \"0fd305cd-b485-40fa-a705-86165c3eadaa\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " Sep 17 18:19:10.161206 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.161101 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rpff\" (UniqueName: \"kubernetes.io/projected/0fd305cd-b485-40fa-a705-86165c3eadaa-kube-api-access-6rpff\") pod \"0fd305cd-b485-40fa-a705-86165c3eadaa\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " Sep 17 18:19:10.161206 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.161122 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd305cd-b485-40fa-a705-86165c3eadaa-istiod-ca-cert\") pod \"0fd305cd-b485-40fa-a705-86165c3eadaa\" (UID: \"0fd305cd-b485-40fa-a705-86165c3eadaa\") " Sep 17 18:19:10.161206 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.161192 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0fd305cd-b485-40fa-a705-86165c3eadaa" (UID: "0fd305cd-b485-40fa-a705-86165c3eadaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:19:10.161428 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.161399 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:19:10.161554 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.161510 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0fd305cd-b485-40fa-a705-86165c3eadaa" (UID: "0fd305cd-b485-40fa-a705-86165c3eadaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:19:10.161634 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.161612 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-data" (OuterVolumeSpecName: "istio-data") pod "0fd305cd-b485-40fa-a705-86165c3eadaa" (UID: "0fd305cd-b485-40fa-a705-86165c3eadaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:19:10.161691 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.161626 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0fd305cd-b485-40fa-a705-86165c3eadaa" (UID: "0fd305cd-b485-40fa-a705-86165c3eadaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:19:10.161691 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.161659 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd305cd-b485-40fa-a705-86165c3eadaa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0fd305cd-b485-40fa-a705-86165c3eadaa" (UID: "0fd305cd-b485-40fa-a705-86165c3eadaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:19:10.163302 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.163275 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0fd305cd-b485-40fa-a705-86165c3eadaa" (UID: "0fd305cd-b485-40fa-a705-86165c3eadaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:19:10.163667 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.163643 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-token" (OuterVolumeSpecName: "istio-token") pod "0fd305cd-b485-40fa-a705-86165c3eadaa" (UID: "0fd305cd-b485-40fa-a705-86165c3eadaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:19:10.163823 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.163803 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0fd305cd-b485-40fa-a705-86165c3eadaa" (UID: "0fd305cd-b485-40fa-a705-86165c3eadaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:19:10.163872 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.163820 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd305cd-b485-40fa-a705-86165c3eadaa-kube-api-access-6rpff" (OuterVolumeSpecName: "kube-api-access-6rpff") pod "0fd305cd-b485-40fa-a705-86165c3eadaa" (UID: "0fd305cd-b485-40fa-a705-86165c3eadaa"). InnerVolumeSpecName "kube-api-access-6rpff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:19:10.262323 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.262300 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:19:10.262323 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.262320 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:19:10.262447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.262330 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:19:10.262447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.262338 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:19:10.262447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.262347 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd305cd-b485-40fa-a705-86165c3eadaa-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:19:10.262447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.262356 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd305cd-b485-40fa-a705-86165c3eadaa-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:19:10.262447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.262364 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rpff\" (UniqueName: \"kubernetes.io/projected/0fd305cd-b485-40fa-a705-86165c3eadaa-kube-api-access-6rpff\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:19:10.262447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:10.262372 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd305cd-b485-40fa-a705-86165c3eadaa-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:19:11.016876 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:11.016846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" event={"ID":"0fd305cd-b485-40fa-a705-86165c3eadaa","Type":"ContainerDied","Data":"34b253782c9ff6f2a49d226041215c73786ea830adb51cec8d10d60724d8a98c"} Sep 17 18:19:11.017233 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:11.016896 2564 scope.go:117] "RemoveContainer" containerID="66d256666113a17636b390ceaed4d2f082dbf21eba4a96202e1976bc97dafc0d" Sep 17 18:19:11.017233 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:11.016858 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk" Sep 17 18:19:11.109354 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:11.109324 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk"] Sep 17 18:19:11.134903 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:11.134880 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-700d65b8-openshift-default-5cc8595d64-l64jk"] Sep 17 18:19:11.452033 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:11.452003 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" path="/var/lib/kubelet/pods/0fd305cd-b485-40fa-a705-86165c3eadaa/volumes" Sep 17 18:19:42.890779 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.890744 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22"] Sep 17 18:19:42.891236 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.891217 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerName="istio-proxy" Sep 17 18:19:42.891278 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.891238 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerName="istio-proxy" Sep 17 18:19:42.891278 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.891250 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6d1e4d1-2093-49ad-ad06-f159a5fbe394" containerName="payload-pre-processing" Sep 17 18:19:42.891278 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.891256 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6d1e4d1-2093-49ad-ad06-f159a5fbe394" containerName="payload-pre-processing" Sep 17 18:19:42.891432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.891278 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62afc752-5ca6-44d5-b288-41e4f13e63a5" containerName="revoke-keys" Sep 17 18:19:42.891432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.891284 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="62afc752-5ca6-44d5-b288-41e4f13e63a5" containerName="revoke-keys" Sep 17 18:19:42.891432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.891292 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c9e7ca-c86f-495e-91aa-f5334dc49d44" containerName="payload-processing" Sep 17 18:19:42.891432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.891299 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c9e7ca-c86f-495e-91aa-f5334dc49d44" containerName="payload-processing" Sep 17 18:19:42.891432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.891361 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="62afc752-5ca6-44d5-b288-41e4f13e63a5" containerName="revoke-keys" Sep 17 18:19:42.891432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.891368 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6d1e4d1-2093-49ad-ad06-f159a5fbe394" containerName="payload-pre-processing" Sep 17 18:19:42.891432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.891376 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerName="istio-proxy" Sep 17 18:19:42.891432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.891382 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c9e7ca-c86f-495e-91aa-f5334dc49d44" containerName="payload-processing" Sep 17 18:19:42.891432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.891387 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerName="istio-proxy" Sep 17 18:19:42.894851 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.894832 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:42.897612 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.897591 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-83507f-openshift-default-dockercfg-tbrbd\"" Sep 17 18:19:42.907201 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.907183 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22"] Sep 17 18:19:42.934112 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.934079 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39910a38-f50b-4067-b424-740a914ccd63-istio-token\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:42.934230 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.934119 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39910a38-f50b-4067-b424-740a914ccd63-istiod-ca-cert\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:42.934298 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.934220 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-workload-certs\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:42.934298 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.934265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39910a38-f50b-4067-b424-740a914ccd63-istio-podinfo\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:42.934407 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.934356 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-istio-envoy\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:42.934407 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.934402 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-workload-socket\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:42.934590 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.934428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-credential-socket\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:42.934590 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.934510 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4ng4\" (UniqueName: \"kubernetes.io/projected/39910a38-f50b-4067-b424-740a914ccd63-kube-api-access-k4ng4\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:42.934590 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:42.934559 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-istio-data\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.035709 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.035671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39910a38-f50b-4067-b424-740a914ccd63-istio-token\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.035915 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.035717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39910a38-f50b-4067-b424-740a914ccd63-istiod-ca-cert\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.035915 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.035863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-workload-certs\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.036044 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.035915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39910a38-f50b-4067-b424-740a914ccd63-istio-podinfo\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.036044 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.035996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-istio-envoy\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.036166 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.036141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-workload-socket\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.036223 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.036203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-credential-socket\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.036278 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.036229 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-workload-certs\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.036335 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.036301 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ng4\" (UniqueName: \"kubernetes.io/projected/39910a38-f50b-4067-b424-740a914ccd63-kube-api-access-k4ng4\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.036393 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.036338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-istio-data\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.036502 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.036481 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-workload-socket\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.036597 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.036499 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-credential-socket\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.036898 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.036877 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-istio-data\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.037295 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.037273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39910a38-f50b-4067-b424-740a914ccd63-istiod-ca-cert\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.038640 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.038623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-istio-envoy\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.038752 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.038730 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39910a38-f50b-4067-b424-740a914ccd63-istio-podinfo\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.048065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.048035 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4ng4\" (UniqueName: \"kubernetes.io/projected/39910a38-f50b-4067-b424-740a914ccd63-kube-api-access-k4ng4\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.048176 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.048155 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39910a38-f50b-4067-b424-740a914ccd63-istio-token\") pod \"e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.206288 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.206218 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:43.491468 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.491437 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22"] Sep 17 18:19:43.492763 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:19:43.492739 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39910a38_f50b_4067_b424_740a914ccd63.slice/crio-045486ff0e37d399c1c194c1d7398d60c54e9cb75c0777198a3335e7f8159a45 WatchSource:0}: Error finding container 045486ff0e37d399c1c194c1d7398d60c54e9cb75c0777198a3335e7f8159a45: Status 404 returned error can't find the container with id 045486ff0e37d399c1c194c1d7398d60c54e9cb75c0777198a3335e7f8159a45 Sep 17 18:19:43.495048 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.495010 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:19:43.495135 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.495081 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:19:43.495135 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:43.495108 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:19:44.222317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:44.222283 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" event={"ID":"39910a38-f50b-4067-b424-740a914ccd63","Type":"ContainerStarted","Data":"232c512f0d29a0fa2dba38abe647b93ca59b599709ddb9fdfeb31e4145d3f352"} Sep 17 18:19:44.222830 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:44.222345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" event={"ID":"39910a38-f50b-4067-b424-740a914ccd63","Type":"ContainerStarted","Data":"045486ff0e37d399c1c194c1d7398d60c54e9cb75c0777198a3335e7f8159a45"} Sep 17 18:19:44.247785 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:44.247735 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" podStartSLOduration=2.247719618 podStartE2EDuration="2.247719618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:44.245093856 +0000 UTC m=+1009.255280992" watchObservedRunningTime="2026-09-17 18:19:44.247719618 +0000 UTC m=+1009.257906754" Sep 17 18:19:45.206498 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:45.206465 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:45.212081 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:45.212059 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:45.228084 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:45.228053 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:45.228900 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:45.228879 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:19:50.831844 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:50.831813 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv"] Sep 17 18:19:50.832249 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:50.832234 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerName="istio-proxy" Sep 17 18:19:50.832293 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:50.832252 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fd305cd-b485-40fa-a705-86165c3eadaa" containerName="istio-proxy" Sep 17 18:19:50.835678 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:50.835661 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" Sep 17 18:19:50.838510 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:50.838486 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-83507f\"" Sep 17 18:19:50.838666 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:50.838496 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-83507f-dockercfg-xx6vq\"" Sep 17 18:19:50.851034 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:50.851011 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv"] Sep 17 18:19:50.910662 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:50.910625 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0906f3f7-0ce0-4622-a0ed-6f784d51eac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv\" (UID: \"0906f3f7-0ce0-4622-a0ed-6f784d51eac9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" Sep 17 18:19:50.911088 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:50.911055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g4z7\" (UniqueName: \"kubernetes.io/projected/0906f3f7-0ce0-4622-a0ed-6f784d51eac9-kube-api-access-6g4z7\") pod \"payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv\" (UID: \"0906f3f7-0ce0-4622-a0ed-6f784d51eac9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" Sep 17 18:19:51.012263 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:51.012223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0906f3f7-0ce0-4622-a0ed-6f784d51eac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv\" (UID: \"0906f3f7-0ce0-4622-a0ed-6f784d51eac9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" Sep 17 18:19:51.012437 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:51.012389 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g4z7\" (UniqueName: \"kubernetes.io/projected/0906f3f7-0ce0-4622-a0ed-6f784d51eac9-kube-api-access-6g4z7\") pod \"payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv\" (UID: \"0906f3f7-0ce0-4622-a0ed-6f784d51eac9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" Sep 17 18:19:51.012992 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:51.012974 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0906f3f7-0ce0-4622-a0ed-6f784d51eac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv\" (UID: \"0906f3f7-0ce0-4622-a0ed-6f784d51eac9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" Sep 17 18:19:51.021421 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:51.021395 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g4z7\" (UniqueName: \"kubernetes.io/projected/0906f3f7-0ce0-4622-a0ed-6f784d51eac9-kube-api-access-6g4z7\") pod \"payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv\" (UID: \"0906f3f7-0ce0-4622-a0ed-6f784d51eac9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" Sep 17 18:19:51.146711 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:51.146636 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" Sep 17 18:19:51.431429 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:51.431402 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv"] Sep 17 18:19:51.432327 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:19:51.432301 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0906f3f7_0ce0_4622_a0ed_6f784d51eac9.slice/crio-ad5ebabd3c9c9e98f43edaabaa8c8928dbbd6c9d4be23775fd353c7ef7a6120d WatchSource:0}: Error finding container ad5ebabd3c9c9e98f43edaabaa8c8928dbbd6c9d4be23775fd353c7ef7a6120d: Status 404 returned error can't find the container with id ad5ebabd3c9c9e98f43edaabaa8c8928dbbd6c9d4be23775fd353c7ef7a6120d Sep 17 18:19:52.281822 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:52.281789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" event={"ID":"0906f3f7-0ce0-4622-a0ed-6f784d51eac9","Type":"ContainerStarted","Data":"e05fbdfe781779890af122e73487cdbbd058d13b6a2d8ec65c13643d8bfaf993"} Sep 17 18:19:52.282204 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:52.281849 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" event={"ID":"0906f3f7-0ce0-4622-a0ed-6f784d51eac9","Type":"ContainerStarted","Data":"ad5ebabd3c9c9e98f43edaabaa8c8928dbbd6c9d4be23775fd353c7ef7a6120d"} Sep 17 18:19:52.282204 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:52.281877 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" Sep 17 18:19:52.299061 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:19:52.299016 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" podStartSLOduration=2.299001055 podStartE2EDuration="2.299001055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:52.297704318 +0000 UTC m=+1017.307891453" watchObservedRunningTime="2026-09-17 18:19:52.299001055 +0000 UTC m=+1017.309188189" Sep 17 18:20:03.290577 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:03.290546 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" Sep 17 18:20:12.410967 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:12.410941 2564 generic.go:356] "Generic (PLEG): container finished" podID="39910a38-f50b-4067-b424-740a914ccd63" containerID="232c512f0d29a0fa2dba38abe647b93ca59b599709ddb9fdfeb31e4145d3f352" exitCode=0 Sep 17 18:20:12.411399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:12.411019 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" event={"ID":"39910a38-f50b-4067-b424-740a914ccd63","Type":"ContainerDied","Data":"232c512f0d29a0fa2dba38abe647b93ca59b599709ddb9fdfeb31e4145d3f352"} Sep 17 18:20:12.411560 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:12.411521 2564 scope.go:117] "RemoveContainer" containerID="232c512f0d29a0fa2dba38abe647b93ca59b599709ddb9fdfeb31e4145d3f352" Sep 17 18:20:12.412810 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:12.412776 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:20:12.412913 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:12.412862 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:20:12.412977 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:12.412915 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:20:13.419443 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:13.419410 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" event={"ID":"39910a38-f50b-4067-b424-740a914ccd63","Type":"ContainerStarted","Data":"74867cd55291c01fd83646e2d9d3ad187be53708333a7422ff90c065bb4469ec"} Sep 17 18:20:14.207447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:14.207411 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:20:14.208345 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:14.208317 2564 patch_prober.go:28] interesting pod/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 17 18:20:14.208471 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:14.208372 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" podUID="39910a38-f50b-4067-b424-740a914ccd63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 17 18:20:15.206680 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:15.206645 2564 patch_prober.go:28] interesting pod/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 17 18:20:15.207161 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:15.206706 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" podUID="39910a38-f50b-4067-b424-740a914ccd63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 17 18:20:15.228079 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:15.228054 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:20:16.207136 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:16.207101 2564 patch_prober.go:28] interesting pod/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 17 18:20:16.207510 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:16.207157 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" podUID="39910a38-f50b-4067-b424-740a914ccd63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 17 18:20:17.207071 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:17.207035 2564 patch_prober.go:28] interesting pod/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 17 18:20:17.207252 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:17.207089 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" podUID="39910a38-f50b-4067-b424-740a914ccd63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 17 18:20:18.207439 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:18.207405 2564 patch_prober.go:28] interesting pod/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 17 18:20:18.207841 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:18.207454 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" podUID="39910a38-f50b-4067-b424-740a914ccd63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 17 18:20:19.211508 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:19.211478 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:20:19.212614 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:19.212592 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:20:41.835009 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:41.834974 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j"] Sep 17 18:20:41.838360 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:41.838341 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:41.841020 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:41.840996 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d62f3adc-openshift-default-dockercfg-bmkrf\"" Sep 17 18:20:41.849572 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:41.849546 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j"] Sep 17 18:20:41.901976 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:41.901951 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-workload-certs\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:41.902082 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:41.901995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69q94\" (UniqueName: \"kubernetes.io/projected/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-kube-api-access-69q94\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:41.902082 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:41.902076 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-podinfo\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:41.902178 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:41.902101 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-data\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:41.902178 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:41.902130 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-envoy\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:41.902178 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:41.902151 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-token\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:41.902286 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:41.902250 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istiod-ca-cert\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:41.902320 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:41.902311 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-workload-socket\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:41.902369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:41.902352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-credential-socket\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.003540 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.003494 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-workload-socket\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.003750 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.003552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-credential-socket\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.003750 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.003583 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-workload-certs\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.003750 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.003621 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69q94\" (UniqueName: \"kubernetes.io/projected/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-kube-api-access-69q94\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.003750 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.003710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-podinfo\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.003750 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.003744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-data\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.004032 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.003784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-envoy\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.004032 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.003806 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-token\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.004032 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.003840 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istiod-ca-cert\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.004032 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.003965 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-workload-socket\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.004032 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.004013 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-credential-socket\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.004276 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.004072 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-data\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.004329 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.004311 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-workload-certs\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.004839 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.004818 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istiod-ca-cert\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.006324 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.006303 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-envoy\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.006648 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.006627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-podinfo\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.012287 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.012263 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-token\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.012443 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.012426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69q94\" (UniqueName: \"kubernetes.io/projected/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-kube-api-access-69q94\") pod \"e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.150279 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.150200 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:42.589180 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.589151 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j"] Sep 17 18:20:42.589896 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:20:42.589867 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5e0dc3_6c93_4328_a0ca_be32f52a5076.slice/crio-384a45416f1956c1cb1c47a65729d7be979616f952d3641326633b41f7d406c8 WatchSource:0}: Error finding container 384a45416f1956c1cb1c47a65729d7be979616f952d3641326633b41f7d406c8: Status 404 returned error can't find the container with id 384a45416f1956c1cb1c47a65729d7be979616f952d3641326633b41f7d406c8 Sep 17 18:20:42.592398 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.592366 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:20:42.592485 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.592429 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:20:42.592485 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.592457 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:20:42.611328 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:42.611302 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" event={"ID":"ad5e0dc3-6c93-4328-a0ca-be32f52a5076","Type":"ContainerStarted","Data":"384a45416f1956c1cb1c47a65729d7be979616f952d3641326633b41f7d406c8"} Sep 17 18:20:43.619028 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:43.618995 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" event={"ID":"ad5e0dc3-6c93-4328-a0ca-be32f52a5076","Type":"ContainerStarted","Data":"385e3fbd1de7b4786f93353dd1b5df685f0c000550a0fa2687e49606c6132c96"} Sep 17 18:20:43.639881 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:43.639825 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" podStartSLOduration=2.6398076120000002 podStartE2EDuration="2.639807612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:43.63941471 +0000 UTC m=+1068.649601872" watchObservedRunningTime="2026-09-17 18:20:43.639807612 +0000 UTC m=+1068.649994750" Sep 17 18:20:44.150637 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:44.150605 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:44.155780 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:44.155756 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:44.625652 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:44.625624 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:44.626571 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:44.626550 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:20:52.943070 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:52.942987 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w"] Sep 17 18:20:52.946618 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:52.946598 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" Sep 17 18:20:52.949630 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:52.949611 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d62f3adc-dockercfg-hmxsh\"" Sep 17 18:20:52.949860 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:52.949845 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d62f3adc\"" Sep 17 18:20:52.960202 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:52.960175 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w"] Sep 17 18:20:53.106733 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:53.106697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w\" (UID: \"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" Sep 17 18:20:53.106905 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:53.106759 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pvnt\" (UniqueName: \"kubernetes.io/projected/4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7-kube-api-access-5pvnt\") pod \"payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w\" (UID: \"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" Sep 17 18:20:53.208264 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:53.208191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pvnt\" (UniqueName: \"kubernetes.io/projected/4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7-kube-api-access-5pvnt\") pod \"payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w\" (UID: \"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" Sep 17 18:20:53.208395 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:53.208309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w\" (UID: \"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" Sep 17 18:20:53.209029 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:53.209010 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w\" (UID: \"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" Sep 17 18:20:53.216335 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:53.216306 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pvnt\" (UniqueName: \"kubernetes.io/projected/4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7-kube-api-access-5pvnt\") pod \"payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w\" (UID: \"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" Sep 17 18:20:53.258019 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:53.257995 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" Sep 17 18:20:53.534238 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:53.534216 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w"] Sep 17 18:20:53.536596 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:20:53.536564 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9fa3d1_9e7d_45ed_ae7a_bdf83074dba7.slice/crio-387407c2a0ee4f4071f406fda504a84200b72d7c2d8e14cae577704b7d64abd0 WatchSource:0}: Error finding container 387407c2a0ee4f4071f406fda504a84200b72d7c2d8e14cae577704b7d64abd0: Status 404 returned error can't find the container with id 387407c2a0ee4f4071f406fda504a84200b72d7c2d8e14cae577704b7d64abd0 Sep 17 18:20:53.681633 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:53.681603 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" event={"ID":"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7","Type":"ContainerStarted","Data":"25183202f28e57d14b303156abe1dbaac386c4e418bb114a1d37075111dc9390"} Sep 17 18:20:53.681759 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:53.681650 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" event={"ID":"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7","Type":"ContainerStarted","Data":"387407c2a0ee4f4071f406fda504a84200b72d7c2d8e14cae577704b7d64abd0"} Sep 17 18:20:53.681759 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:53.681681 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" Sep 17 18:20:53.702894 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:20:53.702842 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" podStartSLOduration=1.702827107 podStartE2EDuration="1.702827107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:53.699711988 +0000 UTC m=+1078.709899125" watchObservedRunningTime="2026-09-17 18:20:53.702827107 +0000 UTC m=+1078.713014242" Sep 17 18:21:04.691275 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:04.691244 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" Sep 17 18:21:12.099034 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:12.098998 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh"] Sep 17 18:21:12.103097 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:12.103079 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh" Sep 17 18:21:12.105801 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:12.105778 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 18:21:12.105928 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:12.105808 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\"" Sep 17 18:21:12.111448 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:12.111386 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh"] Sep 17 18:21:12.164269 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:12.164227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027797d4-718b-4fcf-a7a8-79e401b97ec4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-83507f-6ldvh\" (UID: \"027797d4-718b-4fcf-a7a8-79e401b97ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh" Sep 17 18:21:12.265350 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:12.265318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027797d4-718b-4fcf-a7a8-79e401b97ec4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-83507f-6ldvh\" (UID: \"027797d4-718b-4fcf-a7a8-79e401b97ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh" Sep 17 18:21:12.266004 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:12.265978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027797d4-718b-4fcf-a7a8-79e401b97ec4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-83507f-6ldvh\" (UID: \"027797d4-718b-4fcf-a7a8-79e401b97ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh" Sep 17 18:21:12.415442 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:12.415365 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh" Sep 17 18:21:12.724462 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:12.724438 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh"] Sep 17 18:21:12.726644 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:21:12.726613 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027797d4_718b_4fcf_a7a8_79e401b97ec4.slice/crio-51c215d944ecace448dfe7f07d4fb616c7dba6f26fff5422b1d3ad569d20ad76 WatchSource:0}: Error finding container 51c215d944ecace448dfe7f07d4fb616c7dba6f26fff5422b1d3ad569d20ad76: Status 404 returned error can't find the container with id 51c215d944ecace448dfe7f07d4fb616c7dba6f26fff5422b1d3ad569d20ad76 Sep 17 18:21:12.805591 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:12.805562 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh" event={"ID":"027797d4-718b-4fcf-a7a8-79e401b97ec4","Type":"ContainerStarted","Data":"51c215d944ecace448dfe7f07d4fb616c7dba6f26fff5422b1d3ad569d20ad76"} Sep 17 18:21:13.813385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:13.813353 2564 generic.go:356] "Generic (PLEG): container finished" podID="027797d4-718b-4fcf-a7a8-79e401b97ec4" containerID="f0f6be8e25c624c381df626c03f58e6aa18d9e096a22b04fb9c4eddb484068ea" exitCode=0 Sep 17 18:21:13.813710 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:13.813403 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh" event={"ID":"027797d4-718b-4fcf-a7a8-79e401b97ec4","Type":"ContainerDied","Data":"f0f6be8e25c624c381df626c03f58e6aa18d9e096a22b04fb9c4eddb484068ea"} Sep 17 18:21:15.000636 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:15.000616 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh" Sep 17 18:21:15.096050 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:15.095970 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027797d4-718b-4fcf-a7a8-79e401b97ec4-maas-api-service-ca\") pod \"027797d4-718b-4fcf-a7a8-79e401b97ec4\" (UID: \"027797d4-718b-4fcf-a7a8-79e401b97ec4\") " Sep 17 18:21:15.096304 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:15.096284 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027797d4-718b-4fcf-a7a8-79e401b97ec4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "027797d4-718b-4fcf-a7a8-79e401b97ec4" (UID: "027797d4-718b-4fcf-a7a8-79e401b97ec4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:21:15.197623 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:15.197595 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027797d4-718b-4fcf-a7a8-79e401b97ec4-maas-api-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:15.826789 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:15.826758 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh" Sep 17 18:21:15.826922 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:15.826785 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh" event={"ID":"027797d4-718b-4fcf-a7a8-79e401b97ec4","Type":"ContainerDied","Data":"51c215d944ecace448dfe7f07d4fb616c7dba6f26fff5422b1d3ad569d20ad76"} Sep 17 18:21:15.826922 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:15.826836 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51c215d944ecace448dfe7f07d4fb616c7dba6f26fff5422b1d3ad569d20ad76" Sep 17 18:21:19.861115 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:19.861081 2564 generic.go:356] "Generic (PLEG): container finished" podID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerID="385e3fbd1de7b4786f93353dd1b5df685f0c000550a0fa2687e49606c6132c96" exitCode=0 Sep 17 18:21:19.861664 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:19.861161 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" event={"ID":"ad5e0dc3-6c93-4328-a0ca-be32f52a5076","Type":"ContainerDied","Data":"385e3fbd1de7b4786f93353dd1b5df685f0c000550a0fa2687e49606c6132c96"} Sep 17 18:21:19.861742 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:19.861700 2564 scope.go:117] "RemoveContainer" containerID="385e3fbd1de7b4786f93353dd1b5df685f0c000550a0fa2687e49606c6132c96" Sep 17 18:21:19.862874 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:19.862843 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:21:19.862962 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:19.862909 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:21:19.862962 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:19.862948 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:21:20.869378 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:20.869344 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" event={"ID":"ad5e0dc3-6c93-4328-a0ca-be32f52a5076","Type":"ContainerStarted","Data":"f4517eb1cbd101f017c7691073cbb8b3c81747d6cb88897ded8248352542c87a"} Sep 17 18:21:21.151107 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:21.151015 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:21:21.151432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:21.151408 2564 patch_prober.go:28] interesting pod/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 17 18:21:21.151639 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:21.151603 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" podUID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 17 18:21:22.151345 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:22.150897 2564 patch_prober.go:28] interesting pod/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 17 18:21:22.151345 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:22.150966 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" podUID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 17 18:21:23.151120 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.150661 2564 patch_prober.go:28] interesting pod/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 17 18:21:23.151120 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.150737 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" podUID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 17 18:21:23.428460 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.428372 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv"] Sep 17 18:21:23.428887 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.428716 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" podUID="0906f3f7-0ce0-4622-a0ed-6f784d51eac9" containerName="payload-pre-processing" containerID="cri-o://e05fbdfe781779890af122e73487cdbbd058d13b6a2d8ec65c13643d8bfaf993" gracePeriod=30 Sep 17 18:21:23.598279 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.598247 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh"] Sep 17 18:21:23.600984 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.600961 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-83507f-6ldvh"] Sep 17 18:21:23.747010 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.746698 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" Sep 17 18:21:23.795555 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.789766 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g4z7\" (UniqueName: \"kubernetes.io/projected/0906f3f7-0ce0-4622-a0ed-6f784d51eac9-kube-api-access-6g4z7\") pod \"0906f3f7-0ce0-4622-a0ed-6f784d51eac9\" (UID: \"0906f3f7-0ce0-4622-a0ed-6f784d51eac9\") " Sep 17 18:21:23.795555 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.790348 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0906f3f7-0ce0-4622-a0ed-6f784d51eac9-plugins-config-volume\") pod \"0906f3f7-0ce0-4622-a0ed-6f784d51eac9\" (UID: \"0906f3f7-0ce0-4622-a0ed-6f784d51eac9\") " Sep 17 18:21:23.795555 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.791024 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0906f3f7-0ce0-4622-a0ed-6f784d51eac9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0906f3f7-0ce0-4622-a0ed-6f784d51eac9" (UID: "0906f3f7-0ce0-4622-a0ed-6f784d51eac9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:21:23.806384 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.806352 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0906f3f7-0ce0-4622-a0ed-6f784d51eac9-kube-api-access-6g4z7" (OuterVolumeSpecName: "kube-api-access-6g4z7") pod "0906f3f7-0ce0-4622-a0ed-6f784d51eac9" (UID: "0906f3f7-0ce0-4622-a0ed-6f784d51eac9"). InnerVolumeSpecName "kube-api-access-6g4z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:21:23.891399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.891245 2564 generic.go:356] "Generic (PLEG): container finished" podID="0906f3f7-0ce0-4622-a0ed-6f784d51eac9" containerID="e05fbdfe781779890af122e73487cdbbd058d13b6a2d8ec65c13643d8bfaf993" exitCode=0 Sep 17 18:21:23.891399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.891286 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" Sep 17 18:21:23.891399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.891328 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" event={"ID":"0906f3f7-0ce0-4622-a0ed-6f784d51eac9","Type":"ContainerDied","Data":"e05fbdfe781779890af122e73487cdbbd058d13b6a2d8ec65c13643d8bfaf993"} Sep 17 18:21:23.891399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.891385 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv" event={"ID":"0906f3f7-0ce0-4622-a0ed-6f784d51eac9","Type":"ContainerDied","Data":"ad5ebabd3c9c9e98f43edaabaa8c8928dbbd6c9d4be23775fd353c7ef7a6120d"} Sep 17 18:21:23.891399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.891411 2564 scope.go:117] "RemoveContainer" containerID="e05fbdfe781779890af122e73487cdbbd058d13b6a2d8ec65c13643d8bfaf993" Sep 17 18:21:23.891894 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.891611 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0906f3f7-0ce0-4622-a0ed-6f784d51eac9-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:23.891894 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.891657 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g4z7\" (UniqueName: \"kubernetes.io/projected/0906f3f7-0ce0-4622-a0ed-6f784d51eac9-kube-api-access-6g4z7\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:23.901907 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.901896 2564 scope.go:117] "RemoveContainer" containerID="e05fbdfe781779890af122e73487cdbbd058d13b6a2d8ec65c13643d8bfaf993" Sep 17 18:21:23.902267 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:21:23.902220 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05fbdfe781779890af122e73487cdbbd058d13b6a2d8ec65c13643d8bfaf993\": container with ID starting with e05fbdfe781779890af122e73487cdbbd058d13b6a2d8ec65c13643d8bfaf993 not found: ID does not exist" containerID="e05fbdfe781779890af122e73487cdbbd058d13b6a2d8ec65c13643d8bfaf993" Sep 17 18:21:23.902371 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.902279 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05fbdfe781779890af122e73487cdbbd058d13b6a2d8ec65c13643d8bfaf993"} err="failed to get container status \"e05fbdfe781779890af122e73487cdbbd058d13b6a2d8ec65c13643d8bfaf993\": rpc error: code = NotFound desc = could not find container \"e05fbdfe781779890af122e73487cdbbd058d13b6a2d8ec65c13643d8bfaf993\": container with ID starting with e05fbdfe781779890af122e73487cdbbd058d13b6a2d8ec65c13643d8bfaf993 not found: ID does not exist" Sep 17 18:21:23.914511 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.914490 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv"] Sep 17 18:21:23.918954 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:23.918932 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-83507f-96bbc5b66-rmqsv"] Sep 17 18:21:24.151174 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:24.151138 2564 patch_prober.go:28] interesting pod/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 17 18:21:24.151598 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:24.151562 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" podUID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 17 18:21:25.151022 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:25.150978 2564 patch_prober.go:28] interesting pod/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 17 18:21:25.151516 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:25.151042 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" podUID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 17 18:21:25.471947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:25.471865 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="027797d4-718b-4fcf-a7a8-79e401b97ec4" path="/var/lib/kubelet/pods/027797d4-718b-4fcf-a7a8-79e401b97ec4/volumes" Sep 17 18:21:25.472240 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:25.472227 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0906f3f7-0ce0-4622-a0ed-6f784d51eac9" path="/var/lib/kubelet/pods/0906f3f7-0ce0-4622-a0ed-6f784d51eac9/volumes" Sep 17 18:21:26.155646 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:26.155611 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:21:26.156134 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:26.155991 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:21:26.156575 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:26.156554 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:21:30.822733 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:30.822699 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22"] Sep 17 18:21:30.823104 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:30.823001 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" podUID="39910a38-f50b-4067-b424-740a914ccd63" containerName="istio-proxy" containerID="cri-o://74867cd55291c01fd83646e2d9d3ad187be53708333a7422ff90c065bb4469ec" gracePeriod=30 Sep 17 18:21:34.214113 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:34.214070 2564 patch_prober.go:28] interesting pod/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 18:21:34.214579 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:34.214174 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" podUID="39910a38-f50b-4067-b424-740a914ccd63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:21:35.978837 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:35.978804 2564 generic.go:356] "Generic (PLEG): container finished" podID="39910a38-f50b-4067-b424-740a914ccd63" containerID="74867cd55291c01fd83646e2d9d3ad187be53708333a7422ff90c065bb4469ec" exitCode=0 Sep 17 18:21:35.979192 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:35.978869 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" event={"ID":"39910a38-f50b-4067-b424-740a914ccd63","Type":"ContainerDied","Data":"74867cd55291c01fd83646e2d9d3ad187be53708333a7422ff90c065bb4469ec"} Sep 17 18:21:35.979192 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:35.978938 2564 scope.go:117] "RemoveContainer" containerID="232c512f0d29a0fa2dba38abe647b93ca59b599709ddb9fdfeb31e4145d3f352" Sep 17 18:21:36.090410 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.090357 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:21:36.214445 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.214412 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39910a38-f50b-4067-b424-740a914ccd63-istiod-ca-cert\") pod \"39910a38-f50b-4067-b424-740a914ccd63\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " Sep 17 18:21:36.214608 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.214487 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39910a38-f50b-4067-b424-740a914ccd63-istio-podinfo\") pod \"39910a38-f50b-4067-b424-740a914ccd63\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " Sep 17 18:21:36.214608 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.214593 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-workload-certs\") pod \"39910a38-f50b-4067-b424-740a914ccd63\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " Sep 17 18:21:36.214747 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.214678 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-workload-socket\") pod \"39910a38-f50b-4067-b424-740a914ccd63\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " Sep 17 18:21:36.214747 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.214718 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39910a38-f50b-4067-b424-740a914ccd63-istio-token\") pod \"39910a38-f50b-4067-b424-740a914ccd63\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " Sep 17 18:21:36.214854 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.214757 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-istio-envoy\") pod \"39910a38-f50b-4067-b424-740a914ccd63\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " Sep 17 18:21:36.214854 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.214790 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-istio-data\") pod \"39910a38-f50b-4067-b424-740a914ccd63\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " Sep 17 18:21:36.214854 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.214829 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4ng4\" (UniqueName: \"kubernetes.io/projected/39910a38-f50b-4067-b424-740a914ccd63-kube-api-access-k4ng4\") pod \"39910a38-f50b-4067-b424-740a914ccd63\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " Sep 17 18:21:36.215010 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.214905 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-credential-socket\") pod \"39910a38-f50b-4067-b424-740a914ccd63\" (UID: \"39910a38-f50b-4067-b424-740a914ccd63\") " Sep 17 18:21:36.215897 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.215844 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39910a38-f50b-4067-b424-740a914ccd63" (UID: "39910a38-f50b-4067-b424-740a914ccd63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:21:36.216016 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.215981 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39910a38-f50b-4067-b424-740a914ccd63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39910a38-f50b-4067-b424-740a914ccd63" (UID: "39910a38-f50b-4067-b424-740a914ccd63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:21:36.222569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.222518 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39910a38-f50b-4067-b424-740a914ccd63-istio-token" (OuterVolumeSpecName: "istio-token") pod "39910a38-f50b-4067-b424-740a914ccd63" (UID: "39910a38-f50b-4067-b424-740a914ccd63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:21:36.222696 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.222661 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-istio-data" (OuterVolumeSpecName: "istio-data") pod "39910a38-f50b-4067-b424-740a914ccd63" (UID: "39910a38-f50b-4067-b424-740a914ccd63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:21:36.222696 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.222671 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39910a38-f50b-4067-b424-740a914ccd63" (UID: "39910a38-f50b-4067-b424-740a914ccd63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:21:36.222821 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.222690 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39910a38-f50b-4067-b424-740a914ccd63" (UID: "39910a38-f50b-4067-b424-740a914ccd63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:21:36.223420 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.223393 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39910a38-f50b-4067-b424-740a914ccd63" (UID: "39910a38-f50b-4067-b424-740a914ccd63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:21:36.224650 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.224622 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39910a38-f50b-4067-b424-740a914ccd63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39910a38-f50b-4067-b424-740a914ccd63" (UID: "39910a38-f50b-4067-b424-740a914ccd63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:21:36.224741 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.224715 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39910a38-f50b-4067-b424-740a914ccd63-kube-api-access-k4ng4" (OuterVolumeSpecName: "kube-api-access-k4ng4") pod "39910a38-f50b-4067-b424-740a914ccd63" (UID: "39910a38-f50b-4067-b424-740a914ccd63"). InnerVolumeSpecName "kube-api-access-k4ng4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:21:36.316129 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.316104 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:36.316129 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.316126 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39910a38-f50b-4067-b424-740a914ccd63-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:36.316251 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.316135 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:36.316251 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.316143 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:36.316251 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.316151 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4ng4\" (UniqueName: \"kubernetes.io/projected/39910a38-f50b-4067-b424-740a914ccd63-kube-api-access-k4ng4\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:36.316251 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.316159 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:36.316251 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.316167 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39910a38-f50b-4067-b424-740a914ccd63-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:36.316251 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.316175 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39910a38-f50b-4067-b424-740a914ccd63-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:36.316251 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.316182 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39910a38-f50b-4067-b424-740a914ccd63-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:36.986647 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.986616 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" Sep 17 18:21:36.986647 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.986627 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22" event={"ID":"39910a38-f50b-4067-b424-740a914ccd63","Type":"ContainerDied","Data":"045486ff0e37d399c1c194c1d7398d60c54e9cb75c0777198a3335e7f8159a45"} Sep 17 18:21:36.987159 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:36.986696 2564 scope.go:117] "RemoveContainer" containerID="74867cd55291c01fd83646e2d9d3ad187be53708333a7422ff90c065bb4469ec" Sep 17 18:21:37.009824 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.009799 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22"] Sep 17 18:21:37.013439 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.013419 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-83507f-openshift-default-778bdbb6d8-msg22"] Sep 17 18:21:37.138480 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.137933 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w"] Sep 17 18:21:37.138480 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.138185 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" podUID="4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7" containerName="payload-pre-processing" containerID="cri-o://25183202f28e57d14b303156abe1dbaac386c4e418bb114a1d37075111dc9390" gracePeriod=30 Sep 17 18:21:37.402208 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.402182 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" Sep 17 18:21:37.451947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.451919 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39910a38-f50b-4067-b424-740a914ccd63" path="/var/lib/kubelet/pods/39910a38-f50b-4067-b424-740a914ccd63/volumes" Sep 17 18:21:37.529110 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.528985 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pvnt\" (UniqueName: \"kubernetes.io/projected/4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7-kube-api-access-5pvnt\") pod \"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7\" (UID: \"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7\") " Sep 17 18:21:37.529283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.529179 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7-plugins-config-volume\") pod \"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7\" (UID: \"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7\") " Sep 17 18:21:37.529781 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.529752 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7" (UID: "4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:21:37.529998 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.529979 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:37.533240 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.533216 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7-kube-api-access-5pvnt" (OuterVolumeSpecName: "kube-api-access-5pvnt") pod "4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7" (UID: "4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7"). InnerVolumeSpecName "kube-api-access-5pvnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:21:37.631364 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.631316 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pvnt\" (UniqueName: \"kubernetes.io/projected/4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7-kube-api-access-5pvnt\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:37.999179 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.999143 2564 generic.go:356] "Generic (PLEG): container finished" podID="4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7" containerID="25183202f28e57d14b303156abe1dbaac386c4e418bb114a1d37075111dc9390" exitCode=0 Sep 17 18:21:37.999586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.999189 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" Sep 17 18:21:37.999586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.999211 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" event={"ID":"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7","Type":"ContainerDied","Data":"25183202f28e57d14b303156abe1dbaac386c4e418bb114a1d37075111dc9390"} Sep 17 18:21:37.999586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.999252 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w" event={"ID":"4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7","Type":"ContainerDied","Data":"387407c2a0ee4f4071f406fda504a84200b72d7c2d8e14cae577704b7d64abd0"} Sep 17 18:21:37.999586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:37.999272 2564 scope.go:117] "RemoveContainer" containerID="25183202f28e57d14b303156abe1dbaac386c4e418bb114a1d37075111dc9390" Sep 17 18:21:38.009189 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:38.009172 2564 scope.go:117] "RemoveContainer" containerID="25183202f28e57d14b303156abe1dbaac386c4e418bb114a1d37075111dc9390" Sep 17 18:21:38.009469 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:21:38.009448 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25183202f28e57d14b303156abe1dbaac386c4e418bb114a1d37075111dc9390\": container with ID starting with 25183202f28e57d14b303156abe1dbaac386c4e418bb114a1d37075111dc9390 not found: ID does not exist" containerID="25183202f28e57d14b303156abe1dbaac386c4e418bb114a1d37075111dc9390" Sep 17 18:21:38.009598 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:38.009474 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25183202f28e57d14b303156abe1dbaac386c4e418bb114a1d37075111dc9390"} err="failed to get container status \"25183202f28e57d14b303156abe1dbaac386c4e418bb114a1d37075111dc9390\": rpc error: code = NotFound desc = could not find container \"25183202f28e57d14b303156abe1dbaac386c4e418bb114a1d37075111dc9390\": container with ID starting with 25183202f28e57d14b303156abe1dbaac386c4e418bb114a1d37075111dc9390 not found: ID does not exist" Sep 17 18:21:38.021548 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:38.021512 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w"] Sep 17 18:21:38.027595 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:38.027574 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d62f3adc-65b6cff78b-8n72w"] Sep 17 18:21:39.452012 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:39.451980 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7" path="/var/lib/kubelet/pods/4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7/volumes" Sep 17 18:21:44.130007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:44.129976 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j"] Sep 17 18:21:44.130424 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:44.130179 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" podUID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerName="istio-proxy" containerID="cri-o://f4517eb1cbd101f017c7691073cbb8b3c81747d6cb88897ded8248352542c87a" gracePeriod=30 Sep 17 18:21:49.391764 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.391740 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:21:49.538322 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.538255 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-podinfo\") pod \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " Sep 17 18:21:49.538439 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.538337 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-token\") pod \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " Sep 17 18:21:49.538484 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.538414 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-credential-socket\") pod \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " Sep 17 18:21:49.538484 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.538466 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-data\") pod \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " Sep 17 18:21:49.538582 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.538489 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-workload-certs\") pod \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " Sep 17 18:21:49.538582 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.538505 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-envoy\") pod \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " Sep 17 18:21:49.538582 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.538560 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69q94\" (UniqueName: \"kubernetes.io/projected/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-kube-api-access-69q94\") pod \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " Sep 17 18:21:49.538729 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.538595 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istiod-ca-cert\") pod \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " Sep 17 18:21:49.538729 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.538622 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-workload-socket\") pod \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\" (UID: \"ad5e0dc3-6c93-4328-a0ca-be32f52a5076\") " Sep 17 18:21:49.538729 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.538711 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ad5e0dc3-6c93-4328-a0ca-be32f52a5076" (UID: "ad5e0dc3-6c93-4328-a0ca-be32f52a5076"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:21:49.538729 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.538721 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ad5e0dc3-6c93-4328-a0ca-be32f52a5076" (UID: "ad5e0dc3-6c93-4328-a0ca-be32f52a5076"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:21:49.538936 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.538826 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-data" (OuterVolumeSpecName: "istio-data") pod "ad5e0dc3-6c93-4328-a0ca-be32f52a5076" (UID: "ad5e0dc3-6c93-4328-a0ca-be32f52a5076"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:21:49.539006 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.538975 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ad5e0dc3-6c93-4328-a0ca-be32f52a5076" (UID: "ad5e0dc3-6c93-4328-a0ca-be32f52a5076"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:21:49.539067 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.539003 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ad5e0dc3-6c93-4328-a0ca-be32f52a5076" (UID: "ad5e0dc3-6c93-4328-a0ca-be32f52a5076"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:21:49.539067 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.539032 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:49.539067 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.539053 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:49.539197 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.539069 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:49.540918 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.540891 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-kube-api-access-69q94" (OuterVolumeSpecName: "kube-api-access-69q94") pod "ad5e0dc3-6c93-4328-a0ca-be32f52a5076" (UID: "ad5e0dc3-6c93-4328-a0ca-be32f52a5076"). InnerVolumeSpecName "kube-api-access-69q94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:21:49.540918 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.540891 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ad5e0dc3-6c93-4328-a0ca-be32f52a5076" (UID: "ad5e0dc3-6c93-4328-a0ca-be32f52a5076"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:21:49.541092 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.541078 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad5e0dc3-6c93-4328-a0ca-be32f52a5076" (UID: "ad5e0dc3-6c93-4328-a0ca-be32f52a5076"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:21:49.541129 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.541090 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ad5e0dc3-6c93-4328-a0ca-be32f52a5076" (UID: "ad5e0dc3-6c93-4328-a0ca-be32f52a5076"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:21:49.640414 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.640381 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:49.640414 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.640406 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:49.640414 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.640416 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:49.640682 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.640425 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:49.640682 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.640433 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:49.640682 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:49.640441 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69q94\" (UniqueName: \"kubernetes.io/projected/ad5e0dc3-6c93-4328-a0ca-be32f52a5076-kube-api-access-69q94\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:21:50.080754 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:50.080718 2564 generic.go:356] "Generic (PLEG): container finished" podID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerID="f4517eb1cbd101f017c7691073cbb8b3c81747d6cb88897ded8248352542c87a" exitCode=0 Sep 17 18:21:50.080933 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:50.080754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" event={"ID":"ad5e0dc3-6c93-4328-a0ca-be32f52a5076","Type":"ContainerDied","Data":"f4517eb1cbd101f017c7691073cbb8b3c81747d6cb88897ded8248352542c87a"} Sep 17 18:21:50.080933 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:50.080778 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" Sep 17 18:21:50.080933 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:50.080837 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j" event={"ID":"ad5e0dc3-6c93-4328-a0ca-be32f52a5076","Type":"ContainerDied","Data":"384a45416f1956c1cb1c47a65729d7be979616f952d3641326633b41f7d406c8"} Sep 17 18:21:50.080933 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:50.080860 2564 scope.go:117] "RemoveContainer" containerID="f4517eb1cbd101f017c7691073cbb8b3c81747d6cb88897ded8248352542c87a" Sep 17 18:21:50.091374 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:50.091359 2564 scope.go:117] "RemoveContainer" containerID="385e3fbd1de7b4786f93353dd1b5df685f0c000550a0fa2687e49606c6132c96" Sep 17 18:21:50.099974 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:50.099956 2564 scope.go:117] "RemoveContainer" containerID="f4517eb1cbd101f017c7691073cbb8b3c81747d6cb88897ded8248352542c87a" Sep 17 18:21:50.100223 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:21:50.100204 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4517eb1cbd101f017c7691073cbb8b3c81747d6cb88897ded8248352542c87a\": container with ID starting with f4517eb1cbd101f017c7691073cbb8b3c81747d6cb88897ded8248352542c87a not found: ID does not exist" containerID="f4517eb1cbd101f017c7691073cbb8b3c81747d6cb88897ded8248352542c87a" Sep 17 18:21:50.100286 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:50.100230 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4517eb1cbd101f017c7691073cbb8b3c81747d6cb88897ded8248352542c87a"} err="failed to get container status \"f4517eb1cbd101f017c7691073cbb8b3c81747d6cb88897ded8248352542c87a\": rpc error: code = NotFound desc = could not find container \"f4517eb1cbd101f017c7691073cbb8b3c81747d6cb88897ded8248352542c87a\": container with ID starting with f4517eb1cbd101f017c7691073cbb8b3c81747d6cb88897ded8248352542c87a not found: ID does not exist" Sep 17 18:21:50.100286 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:50.100247 2564 scope.go:117] "RemoveContainer" containerID="385e3fbd1de7b4786f93353dd1b5df685f0c000550a0fa2687e49606c6132c96" Sep 17 18:21:50.100494 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:21:50.100480 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385e3fbd1de7b4786f93353dd1b5df685f0c000550a0fa2687e49606c6132c96\": container with ID starting with 385e3fbd1de7b4786f93353dd1b5df685f0c000550a0fa2687e49606c6132c96 not found: ID does not exist" containerID="385e3fbd1de7b4786f93353dd1b5df685f0c000550a0fa2687e49606c6132c96" Sep 17 18:21:50.100562 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:50.100497 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385e3fbd1de7b4786f93353dd1b5df685f0c000550a0fa2687e49606c6132c96"} err="failed to get container status \"385e3fbd1de7b4786f93353dd1b5df685f0c000550a0fa2687e49606c6132c96\": rpc error: code = NotFound desc = could not find container \"385e3fbd1de7b4786f93353dd1b5df685f0c000550a0fa2687e49606c6132c96\": container with ID starting with 385e3fbd1de7b4786f93353dd1b5df685f0c000550a0fa2687e49606c6132c96 not found: ID does not exist" Sep 17 18:21:50.104919 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:50.104898 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j"] Sep 17 18:21:50.111721 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:50.111688 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d62f3adc-openshift-default-89d5c776d-rjw4j"] Sep 17 18:21:51.451725 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:51.451691 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" path="/var/lib/kubelet/pods/ad5e0dc3-6c93-4328-a0ca-be32f52a5076/volumes" Sep 17 18:21:57.001389 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001358 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb"] Sep 17 18:21:57.001821 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001791 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0906f3f7-0ce0-4622-a0ed-6f784d51eac9" containerName="payload-pre-processing" Sep 17 18:21:57.001821 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001802 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0906f3f7-0ce0-4622-a0ed-6f784d51eac9" containerName="payload-pre-processing" Sep 17 18:21:57.001821 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001811 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7" containerName="payload-pre-processing" Sep 17 18:21:57.001821 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001817 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7" containerName="payload-pre-processing" Sep 17 18:21:57.001956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001829 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39910a38-f50b-4067-b424-740a914ccd63" containerName="istio-proxy" Sep 17 18:21:57.001956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001835 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="39910a38-f50b-4067-b424-740a914ccd63" containerName="istio-proxy" Sep 17 18:21:57.001956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001841 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerName="istio-proxy" Sep 17 18:21:57.001956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001846 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerName="istio-proxy" Sep 17 18:21:57.001956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001863 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027797d4-718b-4fcf-a7a8-79e401b97ec4" containerName="revoke-keys" Sep 17 18:21:57.001956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001868 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="027797d4-718b-4fcf-a7a8-79e401b97ec4" containerName="revoke-keys" Sep 17 18:21:57.001956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001882 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39910a38-f50b-4067-b424-740a914ccd63" containerName="istio-proxy" Sep 17 18:21:57.001956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001886 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="39910a38-f50b-4067-b424-740a914ccd63" containerName="istio-proxy" Sep 17 18:21:57.001956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001956 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9fa3d1-9e7d-45ed-ae7a-bdf83074dba7" containerName="payload-pre-processing" Sep 17 18:21:57.002261 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001965 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="39910a38-f50b-4067-b424-740a914ccd63" containerName="istio-proxy" Sep 17 18:21:57.002261 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001971 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerName="istio-proxy" Sep 17 18:21:57.002261 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001976 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="39910a38-f50b-4067-b424-740a914ccd63" containerName="istio-proxy" Sep 17 18:21:57.002261 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001982 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerName="istio-proxy" Sep 17 18:21:57.002261 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001989 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="027797d4-718b-4fcf-a7a8-79e401b97ec4" containerName="revoke-keys" Sep 17 18:21:57.002261 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.001994 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0906f3f7-0ce0-4622-a0ed-6f784d51eac9" containerName="payload-pre-processing" Sep 17 18:21:57.006667 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.006651 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.010168 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.010146 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"kube-root-ca.crt\"" Sep 17 18:21:57.010295 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.010163 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"distinct-w0-5d8053-kserve-self-signed-certs\"" Sep 17 18:21:57.010295 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.010181 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"openshift-service-ca.crt\"" Sep 17 18:21:57.010295 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.010265 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"default-dockercfg-t6hzs\"" Sep 17 18:21:57.015436 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.015416 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb"] Sep 17 18:21:57.104852 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.104827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-tls-certs\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.105014 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.104857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-home\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.105014 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.104872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-dshm\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.105014 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.104905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-tmp-dir\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.105014 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.105001 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjp6s\" (UniqueName: \"kubernetes.io/projected/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-kube-api-access-mjp6s\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.105220 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.105095 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-model-cache\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.206157 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.206132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-model-cache\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.206328 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.206178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-tls-certs\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.206328 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.206196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-home\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.206460 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.206377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-dshm\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.206519 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.206463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-tmp-dir\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.206618 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.206567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjp6s\" (UniqueName: \"kubernetes.io/projected/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-kube-api-access-mjp6s\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.206674 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.206619 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-model-cache\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.206674 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.206656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-home\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.206788 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.206776 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-tmp-dir\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.208815 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.208789 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-dshm\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.209052 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.209034 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-tls-certs\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.214891 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.214869 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjp6s\" (UniqueName: \"kubernetes.io/projected/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-kube-api-access-mjp6s\") pod \"distinct-w0-5d8053-kserve-865984d9f8-5t5hb\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.316423 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.316401 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:21:57.619218 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:57.619191 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb"] Sep 17 18:21:57.620066 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:21:57.620035 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb14b32c9_c9df_4054_a0bd_6f9a9dc676e2.slice/crio-32b38c674bd98a7607dfbefc01e1ef3bde65529748c91eb9c32ebaa0ec9d3a25 WatchSource:0}: Error finding container 32b38c674bd98a7607dfbefc01e1ef3bde65529748c91eb9c32ebaa0ec9d3a25: Status 404 returned error can't find the container with id 32b38c674bd98a7607dfbefc01e1ef3bde65529748c91eb9c32ebaa0ec9d3a25 Sep 17 18:21:58.133149 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.133115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" event={"ID":"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2","Type":"ContainerStarted","Data":"296c94ea45793c73071c86bbeb2e7efef09a8a3b55ec8f7c514aa1aafe4788b2"} Sep 17 18:21:58.133516 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.133155 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" event={"ID":"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2","Type":"ContainerStarted","Data":"32b38c674bd98a7607dfbefc01e1ef3bde65529748c91eb9c32ebaa0ec9d3a25"} Sep 17 18:21:58.158452 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.158396 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" podStartSLOduration=2.158378659 podStartE2EDuration="2.158378659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:58.157830026 +0000 UTC m=+1143.168017163" watchObservedRunningTime="2026-09-17 18:21:58.158378659 +0000 UTC m=+1143.168565797" Sep 17 18:21:58.524764 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.524670 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847"] Sep 17 18:21:58.525142 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.525126 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerName="istio-proxy" Sep 17 18:21:58.525203 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.525144 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5e0dc3-6c93-4328-a0ca-be32f52a5076" containerName="istio-proxy" Sep 17 18:21:58.528618 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.528601 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.530930 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.530908 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-1a9245-openshift-default-dockercfg-8gd8v\"" Sep 17 18:21:58.541743 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.541721 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847"] Sep 17 18:21:58.620995 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.620698 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istiod-ca-cert\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.620995 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.620783 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-credential-socket\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.620995 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.620814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-workload-socket\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.620995 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.620842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-envoy\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.620995 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.620883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-data\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.620995 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.620913 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-token\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.620995 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.620952 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-podinfo\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.620995 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.620981 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f22vm\" (UniqueName: \"kubernetes.io/projected/c637b203-8705-4b9b-b5b5-9fa6728be0b8-kube-api-access-f22vm\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.621597 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.621043 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-workload-certs\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.721738 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.721698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istiod-ca-cert\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.721934 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.721771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-credential-socket\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.721934 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.721804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-workload-socket\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.721934 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.721829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-envoy\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.722114 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.721976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-data\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.722114 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.722035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-token\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.722114 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.722081 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-podinfo\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.722114 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.722109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f22vm\" (UniqueName: \"kubernetes.io/projected/c637b203-8705-4b9b-b5b5-9fa6728be0b8-kube-api-access-f22vm\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.722325 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.722188 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-workload-certs\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.722325 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.722282 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-credential-socket\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.722431 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.722415 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-workload-socket\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.722560 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.722507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-workload-certs\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.722560 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.722550 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istiod-ca-cert\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.722873 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.722562 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-data\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.724303 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.724272 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-envoy\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.724725 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.724706 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-podinfo\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.730429 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.730399 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-token\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.730713 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.730696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f22vm\" (UniqueName: \"kubernetes.io/projected/c637b203-8705-4b9b-b5b5-9fa6728be0b8-kube-api-access-f22vm\") pod \"e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:58.842568 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:58.842547 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:21:59.122849 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:59.122825 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847"] Sep 17 18:21:59.124005 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:21:59.123981 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc637b203_8705_4b9b_b5b5_9fa6728be0b8.slice/crio-1daae948eb93d88f71937f73d318fa8042d4bad0536b9421256634b9ea1f1058 WatchSource:0}: Error finding container 1daae948eb93d88f71937f73d318fa8042d4bad0536b9421256634b9ea1f1058: Status 404 returned error can't find the container with id 1daae948eb93d88f71937f73d318fa8042d4bad0536b9421256634b9ea1f1058 Sep 17 18:21:59.126317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:59.126284 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:21:59.126382 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:59.126348 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:21:59.126382 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:59.126376 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:21:59.140793 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:21:59.140767 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" event={"ID":"c637b203-8705-4b9b-b5b5-9fa6728be0b8","Type":"ContainerStarted","Data":"1daae948eb93d88f71937f73d318fa8042d4bad0536b9421256634b9ea1f1058"} Sep 17 18:22:00.148739 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:00.148701 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" event={"ID":"c637b203-8705-4b9b-b5b5-9fa6728be0b8","Type":"ContainerStarted","Data":"354906a9559eaac75d70493c89d4ca901bf459abc11ce43fad7713bf6ce20572"} Sep 17 18:22:00.168848 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:00.168807 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" podStartSLOduration=2.168793264 podStartE2EDuration="2.168793264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:00.16866123 +0000 UTC m=+1145.178848365" watchObservedRunningTime="2026-09-17 18:22:00.168793264 +0000 UTC m=+1145.178980399" Sep 17 18:22:00.842999 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:00.842966 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:22:00.848220 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:00.848196 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:22:01.154792 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:01.154727 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:22:01.155652 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:01.155622 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:22:06.043897 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.043868 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt"] Sep 17 18:22:06.047665 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.047649 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" Sep 17 18:22:06.050092 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.050063 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-1a9245\"" Sep 17 18:22:06.056988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.056961 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt"] Sep 17 18:22:06.084300 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.084273 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7"] Sep 17 18:22:06.087634 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.087612 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" Sep 17 18:22:06.089822 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.089805 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-1a9245-dockercfg-r9kr6\"" Sep 17 18:22:06.090036 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.090016 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx8pd\" (UniqueName: \"kubernetes.io/projected/685be49a-b0ef-4c07-9347-a29acabe48d3-kube-api-access-tx8pd\") pod \"payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt\" (UID: \"685be49a-b0ef-4c07-9347-a29acabe48d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" Sep 17 18:22:06.090147 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.090125 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685be49a-b0ef-4c07-9347-a29acabe48d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt\" (UID: \"685be49a-b0ef-4c07-9347-a29acabe48d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" Sep 17 18:22:06.097222 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.097201 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7"] Sep 17 18:22:06.190840 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.190803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqcbp\" (UniqueName: \"kubernetes.io/projected/a253f7c8-24c5-4caf-b6d0-98127fd0d902-kube-api-access-kqcbp\") pod \"payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7\" (UID: \"a253f7c8-24c5-4caf-b6d0-98127fd0d902\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" Sep 17 18:22:06.191239 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.191213 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx8pd\" (UniqueName: \"kubernetes.io/projected/685be49a-b0ef-4c07-9347-a29acabe48d3-kube-api-access-tx8pd\") pod \"payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt\" (UID: \"685be49a-b0ef-4c07-9347-a29acabe48d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" Sep 17 18:22:06.191907 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.191883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a253f7c8-24c5-4caf-b6d0-98127fd0d902-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7\" (UID: \"a253f7c8-24c5-4caf-b6d0-98127fd0d902\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" Sep 17 18:22:06.192300 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.192282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685be49a-b0ef-4c07-9347-a29acabe48d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt\" (UID: \"685be49a-b0ef-4c07-9347-a29acabe48d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" Sep 17 18:22:06.193332 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.193309 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685be49a-b0ef-4c07-9347-a29acabe48d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt\" (UID: \"685be49a-b0ef-4c07-9347-a29acabe48d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" Sep 17 18:22:06.204486 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.204454 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx8pd\" (UniqueName: \"kubernetes.io/projected/685be49a-b0ef-4c07-9347-a29acabe48d3-kube-api-access-tx8pd\") pod \"payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt\" (UID: \"685be49a-b0ef-4c07-9347-a29acabe48d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" Sep 17 18:22:06.293116 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.293084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a253f7c8-24c5-4caf-b6d0-98127fd0d902-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7\" (UID: \"a253f7c8-24c5-4caf-b6d0-98127fd0d902\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" Sep 17 18:22:06.293281 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.293163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqcbp\" (UniqueName: \"kubernetes.io/projected/a253f7c8-24c5-4caf-b6d0-98127fd0d902-kube-api-access-kqcbp\") pod \"payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7\" (UID: \"a253f7c8-24c5-4caf-b6d0-98127fd0d902\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" Sep 17 18:22:06.293804 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.293784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a253f7c8-24c5-4caf-b6d0-98127fd0d902-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7\" (UID: \"a253f7c8-24c5-4caf-b6d0-98127fd0d902\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" Sep 17 18:22:06.302169 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.302106 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqcbp\" (UniqueName: \"kubernetes.io/projected/a253f7c8-24c5-4caf-b6d0-98127fd0d902-kube-api-access-kqcbp\") pod \"payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7\" (UID: \"a253f7c8-24c5-4caf-b6d0-98127fd0d902\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" Sep 17 18:22:06.360478 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.360454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" Sep 17 18:22:06.398208 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.398183 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" Sep 17 18:22:06.635911 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.635883 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt"] Sep 17 18:22:06.636461 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:22:06.636436 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685be49a_b0ef_4c07_9347_a29acabe48d3.slice/crio-8f44bce38d88b4fd5030ab40a57c9a7f841f3a8db7a0fcbc90a613d33c48a34a WatchSource:0}: Error finding container 8f44bce38d88b4fd5030ab40a57c9a7f841f3a8db7a0fcbc90a613d33c48a34a: Status 404 returned error can't find the container with id 8f44bce38d88b4fd5030ab40a57c9a7f841f3a8db7a0fcbc90a613d33c48a34a Sep 17 18:22:06.683493 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:06.683471 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7"] Sep 17 18:22:06.689501 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:22:06.689475 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda253f7c8_24c5_4caf_b6d0_98127fd0d902.slice/crio-8b419f314c048555b4652644ea59b94dae679790e8fd745cf5127b8c09c45f7c WatchSource:0}: Error finding container 8b419f314c048555b4652644ea59b94dae679790e8fd745cf5127b8c09c45f7c: Status 404 returned error can't find the container with id 8b419f314c048555b4652644ea59b94dae679790e8fd745cf5127b8c09c45f7c Sep 17 18:22:07.200859 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:07.200824 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" event={"ID":"a253f7c8-24c5-4caf-b6d0-98127fd0d902","Type":"ContainerStarted","Data":"959ba43e07db7d501159ab7ac4d60fdbb2808587a4d3baba5661ee2b3e93d1ac"} Sep 17 18:22:07.201355 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:07.201184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" event={"ID":"a253f7c8-24c5-4caf-b6d0-98127fd0d902","Type":"ContainerStarted","Data":"8b419f314c048555b4652644ea59b94dae679790e8fd745cf5127b8c09c45f7c"} Sep 17 18:22:07.201355 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:07.201231 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" Sep 17 18:22:07.202807 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:07.202776 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" event={"ID":"685be49a-b0ef-4c07-9347-a29acabe48d3","Type":"ContainerStarted","Data":"3ec03b095cd91429e746834df134376400ae181d666372a1fba2b5689c9347c2"} Sep 17 18:22:07.202915 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:07.202830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" event={"ID":"685be49a-b0ef-4c07-9347-a29acabe48d3","Type":"ContainerStarted","Data":"8f44bce38d88b4fd5030ab40a57c9a7f841f3a8db7a0fcbc90a613d33c48a34a"} Sep 17 18:22:07.202915 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:07.202875 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" Sep 17 18:22:07.220038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:07.219983 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" podStartSLOduration=1.219965011 podStartE2EDuration="1.219965011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:07.217792348 +0000 UTC m=+1152.227979485" watchObservedRunningTime="2026-09-17 18:22:07.219965011 +0000 UTC m=+1152.230152148" Sep 17 18:22:07.236263 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:07.236222 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" podStartSLOduration=1.2362110099999999 podStartE2EDuration="1.23621101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:07.234725975 +0000 UTC m=+1152.244913117" watchObservedRunningTime="2026-09-17 18:22:07.23621101 +0000 UTC m=+1152.246398146" Sep 17 18:22:07.317326 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:07.317301 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:22:07.317455 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:07.317348 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:22:07.341092 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:07.338119 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:22:08.221236 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:08.221206 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:22:18.212099 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:18.211990 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" Sep 17 18:22:18.212701 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:18.212679 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" Sep 17 18:22:32.196706 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:32.196664 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 17 18:22:32.197188 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:32.196732 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 18:22:33.364804 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:33.364771 2564 generic.go:356] "Generic (PLEG): container finished" podID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerID="354906a9559eaac75d70493c89d4ca901bf459abc11ce43fad7713bf6ce20572" exitCode=0 Sep 17 18:22:33.365257 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:33.364840 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" event={"ID":"c637b203-8705-4b9b-b5b5-9fa6728be0b8","Type":"ContainerDied","Data":"354906a9559eaac75d70493c89d4ca901bf459abc11ce43fad7713bf6ce20572"} Sep 17 18:22:33.365309 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:33.365272 2564 scope.go:117] "RemoveContainer" containerID="354906a9559eaac75d70493c89d4ca901bf459abc11ce43fad7713bf6ce20572" Sep 17 18:22:33.366420 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:33.366393 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:22:33.366477 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:33.366462 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:22:33.366512 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:33.366492 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:22:34.373968 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:34.373925 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" event={"ID":"c637b203-8705-4b9b-b5b5-9fa6728be0b8","Type":"ContainerStarted","Data":"3afefff311469a3f2943b436a202c3d6ed6d79509aec6da7174ea2049ab88e4d"} Sep 17 18:22:34.843981 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:34.843945 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:22:34.844255 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:34.844225 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 17 18:22:34.844403 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:34.844380 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 17 18:22:35.843341 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:35.843297 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 17 18:22:35.843850 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:35.843362 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 17 18:22:36.843806 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:36.843761 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 17 18:22:36.844293 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:36.843830 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 17 18:22:37.843818 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:37.843774 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 17 18:22:37.844297 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:37.843843 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 17 18:22:38.843681 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:38.843641 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 17 18:22:38.843883 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:38.843709 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 17 18:22:39.846499 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:39.846472 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:22:39.846963 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:39.846747 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:22:39.847487 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:39.847471 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:22:40.617992 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.617959 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87"] Sep 17 18:22:40.621574 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.621522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" Sep 17 18:22:40.624041 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.624020 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-03270e\"" Sep 17 18:22:40.624161 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.624080 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-03270e-dockercfg-wlxw8\"" Sep 17 18:22:40.630844 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.630823 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87"] Sep 17 18:22:40.658778 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.658756 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv"] Sep 17 18:22:40.661241 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.661224 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" Sep 17 18:22:40.675232 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.675197 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv"] Sep 17 18:22:40.720220 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.720195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99kx\" (UniqueName: \"kubernetes.io/projected/7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da-kube-api-access-z99kx\") pod \"payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87\" (UID: \"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" Sep 17 18:22:40.720349 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.720230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87\" (UID: \"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" Sep 17 18:22:40.821460 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.821418 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z99kx\" (UniqueName: \"kubernetes.io/projected/7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da-kube-api-access-z99kx\") pod \"payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87\" (UID: \"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" Sep 17 18:22:40.821646 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.821472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87\" (UID: \"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" Sep 17 18:22:40.821646 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.821599 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62d8345-fe5a-49c5-9360-60f50fca71eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv\" (UID: \"e62d8345-fe5a-49c5-9360-60f50fca71eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" Sep 17 18:22:40.821756 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.821653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwpg\" (UniqueName: \"kubernetes.io/projected/e62d8345-fe5a-49c5-9360-60f50fca71eb-kube-api-access-qtwpg\") pod \"payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv\" (UID: \"e62d8345-fe5a-49c5-9360-60f50fca71eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" Sep 17 18:22:40.822234 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.822212 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87\" (UID: \"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" Sep 17 18:22:40.829773 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.829744 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99kx\" (UniqueName: \"kubernetes.io/projected/7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da-kube-api-access-z99kx\") pod \"payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87\" (UID: \"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" Sep 17 18:22:40.922859 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.922797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62d8345-fe5a-49c5-9360-60f50fca71eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv\" (UID: \"e62d8345-fe5a-49c5-9360-60f50fca71eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" Sep 17 18:22:40.922859 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.922840 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwpg\" (UniqueName: \"kubernetes.io/projected/e62d8345-fe5a-49c5-9360-60f50fca71eb-kube-api-access-qtwpg\") pod \"payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv\" (UID: \"e62d8345-fe5a-49c5-9360-60f50fca71eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" Sep 17 18:22:40.923510 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.923492 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62d8345-fe5a-49c5-9360-60f50fca71eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv\" (UID: \"e62d8345-fe5a-49c5-9360-60f50fca71eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" Sep 17 18:22:40.931262 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.931242 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwpg\" (UniqueName: \"kubernetes.io/projected/e62d8345-fe5a-49c5-9360-60f50fca71eb-kube-api-access-qtwpg\") pod \"payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv\" (UID: \"e62d8345-fe5a-49c5-9360-60f50fca71eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" Sep 17 18:22:40.932575 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.932554 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" Sep 17 18:22:40.972461 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:40.972243 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" Sep 17 18:22:41.209081 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:41.209052 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87"] Sep 17 18:22:41.209298 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:22:41.209275 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6d4d5e_9cab_4be7_b62e_6bb7b873d0da.slice/crio-09f836ce7f53ee50e10d5fe6f2266ae1144527afec93812582cca3f79faa2667 WatchSource:0}: Error finding container 09f836ce7f53ee50e10d5fe6f2266ae1144527afec93812582cca3f79faa2667: Status 404 returned error can't find the container with id 09f836ce7f53ee50e10d5fe6f2266ae1144527afec93812582cca3f79faa2667 Sep 17 18:22:41.262370 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:41.262347 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv"] Sep 17 18:22:41.264091 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:22:41.264069 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62d8345_fe5a_49c5_9360_60f50fca71eb.slice/crio-ca47eba6f2a0b416e18a6712a3dad2d9f95fc9d6d3b59ed2eceb10d8f5ee5e9e WatchSource:0}: Error finding container ca47eba6f2a0b416e18a6712a3dad2d9f95fc9d6d3b59ed2eceb10d8f5ee5e9e: Status 404 returned error can't find the container with id ca47eba6f2a0b416e18a6712a3dad2d9f95fc9d6d3b59ed2eceb10d8f5ee5e9e Sep 17 18:22:41.419720 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:41.419685 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" event={"ID":"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da","Type":"ContainerStarted","Data":"dcb5097189c8306b0e8c99abc577850d532879f7e7b7d23d51b52b101f0dd1a8"} Sep 17 18:22:41.419887 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:41.419748 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" event={"ID":"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da","Type":"ContainerStarted","Data":"09f836ce7f53ee50e10d5fe6f2266ae1144527afec93812582cca3f79faa2667"} Sep 17 18:22:41.419887 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:41.419875 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" Sep 17 18:22:41.421521 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:41.421496 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" event={"ID":"e62d8345-fe5a-49c5-9360-60f50fca71eb","Type":"ContainerStarted","Data":"85bec18417df1721b68606dd27d297dc0c752594c8a1eee50657afaf9f90b283"} Sep 17 18:22:41.421644 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:41.421567 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" event={"ID":"e62d8345-fe5a-49c5-9360-60f50fca71eb","Type":"ContainerStarted","Data":"ca47eba6f2a0b416e18a6712a3dad2d9f95fc9d6d3b59ed2eceb10d8f5ee5e9e"} Sep 17 18:22:41.421720 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:41.421709 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" Sep 17 18:22:41.436882 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:41.436839 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" podStartSLOduration=1.436823923 podStartE2EDuration="1.436823923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:41.435497798 +0000 UTC m=+1186.445684935" watchObservedRunningTime="2026-09-17 18:22:41.436823923 +0000 UTC m=+1186.447011058" Sep 17 18:22:41.453569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:41.453518 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" podStartSLOduration=1.4535056210000001 podStartE2EDuration="1.453505621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:41.451457691 +0000 UTC m=+1186.461644828" watchObservedRunningTime="2026-09-17 18:22:41.453505621 +0000 UTC m=+1186.463692756" Sep 17 18:22:52.429748 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:52.429712 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" Sep 17 18:22:52.430200 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:52.429828 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" Sep 17 18:22:55.570190 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:22:55.570162 2564 scope.go:117] "RemoveContainer" containerID="e7777cdeccff2cc15ef4d3589c72a9685c6c8b88c4b91a10abd14cd7eb30b4aa" Sep 17 18:23:20.074110 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:20.074077 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj"] Sep 17 18:23:20.074555 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:20.074326 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerName="istio-proxy" containerID="cri-o://d615120c5ed6704874b8902b3a04c9d8c573c1634f63e7cc26e88ee4451a81db" gracePeriod=30 Sep 17 18:23:21.976783 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:21.976749 2564 patch_prober.go:28] interesting pod/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 18:23:21.977159 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:21.976806 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:23:25.330357 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.330333 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:23:25.412873 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.412792 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-podinfo\") pod \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " Sep 17 18:23:25.412873 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.412842 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-data\") pod \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " Sep 17 18:23:25.413064 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.412886 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-credential-socket\") pod \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " Sep 17 18:23:25.413064 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.412931 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-token\") pod \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " Sep 17 18:23:25.413519 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.413422 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-envoy\") pod \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " Sep 17 18:23:25.413519 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.413450 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" (UID: "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:23:25.413519 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.413456 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-data" (OuterVolumeSpecName: "istio-data") pod "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" (UID: "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:23:25.413519 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.413493 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-workload-socket\") pod \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " Sep 17 18:23:25.414183 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.413588 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsnm9\" (UniqueName: \"kubernetes.io/projected/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-kube-api-access-lsnm9\") pod \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " Sep 17 18:23:25.414183 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.413618 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istiod-ca-cert\") pod \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " Sep 17 18:23:25.414183 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.413644 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-workload-certs\") pod \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\" (UID: \"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de\") " Sep 17 18:23:25.414183 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.413849 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" (UID: "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:23:25.414183 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.414004 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" (UID: "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:23:25.414183 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.414010 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" (UID: "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:23:25.414183 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.414074 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:23:25.414183 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.414095 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:23:25.414183 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.414109 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:23:25.415829 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.415807 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" (UID: "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:23:25.415960 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.415940 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-token" (OuterVolumeSpecName: "istio-token") pod "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" (UID: "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:23:25.416335 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.416310 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" (UID: "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:23:25.416335 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.416326 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-kube-api-access-lsnm9" (OuterVolumeSpecName: "kube-api-access-lsnm9") pod "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" (UID: "26e8655d-b888-4cf2-b25a-e1b1eb9ab8de"). InnerVolumeSpecName "kube-api-access-lsnm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:23:25.515606 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.515585 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:23:25.515773 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.515758 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:23:25.515885 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.515871 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsnm9\" (UniqueName: \"kubernetes.io/projected/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-kube-api-access-lsnm9\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:23:25.515984 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.515971 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:23:25.516108 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.516094 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:23:25.516273 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.516255 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:23:25.708340 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.708244 2564 generic.go:356] "Generic (PLEG): container finished" podID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerID="d615120c5ed6704874b8902b3a04c9d8c573c1634f63e7cc26e88ee4451a81db" exitCode=0 Sep 17 18:23:25.708340 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.708292 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" Sep 17 18:23:25.708340 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.708328 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" event={"ID":"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de","Type":"ContainerDied","Data":"d615120c5ed6704874b8902b3a04c9d8c573c1634f63e7cc26e88ee4451a81db"} Sep 17 18:23:25.708581 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.708387 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj" event={"ID":"26e8655d-b888-4cf2-b25a-e1b1eb9ab8de","Type":"ContainerDied","Data":"07318f3fd5f4767969f2ca67a126173c664500cd5bbbd00ec7cdc907781b58bf"} Sep 17 18:23:25.708581 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.708410 2564 scope.go:117] "RemoveContainer" containerID="d615120c5ed6704874b8902b3a04c9d8c573c1634f63e7cc26e88ee4451a81db" Sep 17 18:23:25.718391 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.718371 2564 scope.go:117] "RemoveContainer" containerID="f7cd670d197ec8bf4796b98b55dbe1f64cafd80ab3d9aadee756be1fd05b4c7d" Sep 17 18:23:25.727013 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.726941 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj"] Sep 17 18:23:25.727824 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.727805 2564 scope.go:117] "RemoveContainer" containerID="d615120c5ed6704874b8902b3a04c9d8c573c1634f63e7cc26e88ee4451a81db" Sep 17 18:23:25.728167 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:23:25.728143 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d615120c5ed6704874b8902b3a04c9d8c573c1634f63e7cc26e88ee4451a81db\": container with ID starting with d615120c5ed6704874b8902b3a04c9d8c573c1634f63e7cc26e88ee4451a81db not found: ID does not exist" containerID="d615120c5ed6704874b8902b3a04c9d8c573c1634f63e7cc26e88ee4451a81db" Sep 17 18:23:25.728240 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.728179 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d615120c5ed6704874b8902b3a04c9d8c573c1634f63e7cc26e88ee4451a81db"} err="failed to get container status \"d615120c5ed6704874b8902b3a04c9d8c573c1634f63e7cc26e88ee4451a81db\": rpc error: code = NotFound desc = could not find container \"d615120c5ed6704874b8902b3a04c9d8c573c1634f63e7cc26e88ee4451a81db\": container with ID starting with d615120c5ed6704874b8902b3a04c9d8c573c1634f63e7cc26e88ee4451a81db not found: ID does not exist" Sep 17 18:23:25.728240 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.728202 2564 scope.go:117] "RemoveContainer" containerID="f7cd670d197ec8bf4796b98b55dbe1f64cafd80ab3d9aadee756be1fd05b4c7d" Sep 17 18:23:25.728501 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:23:25.728482 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7cd670d197ec8bf4796b98b55dbe1f64cafd80ab3d9aadee756be1fd05b4c7d\": container with ID starting with f7cd670d197ec8bf4796b98b55dbe1f64cafd80ab3d9aadee756be1fd05b4c7d not found: ID does not exist" containerID="f7cd670d197ec8bf4796b98b55dbe1f64cafd80ab3d9aadee756be1fd05b4c7d" Sep 17 18:23:25.728596 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.728510 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7cd670d197ec8bf4796b98b55dbe1f64cafd80ab3d9aadee756be1fd05b4c7d"} err="failed to get container status \"f7cd670d197ec8bf4796b98b55dbe1f64cafd80ab3d9aadee756be1fd05b4c7d\": rpc error: code = NotFound desc = could not find container \"f7cd670d197ec8bf4796b98b55dbe1f64cafd80ab3d9aadee756be1fd05b4c7d\": container with ID starting with f7cd670d197ec8bf4796b98b55dbe1f64cafd80ab3d9aadee756be1fd05b4c7d not found: ID does not exist" Sep 17 18:23:25.730860 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:25.730841 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c31adf-openshift-default-8597b8b85d-6g7pj"] Sep 17 18:23:27.451796 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:27.451764 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" path="/var/lib/kubelet/pods/26e8655d-b888-4cf2-b25a-e1b1eb9ab8de/volumes" Sep 17 18:23:31.163677 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.163645 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr"] Sep 17 18:23:31.164141 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.163892 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" podUID="e9eb58fe-4a3a-4b5b-b050-583b1948236b" containerName="payload-pre-processing" containerID="cri-o://f0329219d6ae19f7c14ecaed764f8cdc803d1e1311f91a99ee38bf1c5ac47994" gracePeriod=30 Sep 17 18:23:31.454025 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.453996 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" Sep 17 18:23:31.570586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.570552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9eb58fe-4a3a-4b5b-b050-583b1948236b-plugins-config-volume\") pod \"e9eb58fe-4a3a-4b5b-b050-583b1948236b\" (UID: \"e9eb58fe-4a3a-4b5b-b050-583b1948236b\") " Sep 17 18:23:31.570765 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.570612 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh9hj\" (UniqueName: \"kubernetes.io/projected/e9eb58fe-4a3a-4b5b-b050-583b1948236b-kube-api-access-bh9hj\") pod \"e9eb58fe-4a3a-4b5b-b050-583b1948236b\" (UID: \"e9eb58fe-4a3a-4b5b-b050-583b1948236b\") " Sep 17 18:23:31.570911 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.570887 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9eb58fe-4a3a-4b5b-b050-583b1948236b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9eb58fe-4a3a-4b5b-b050-583b1948236b" (UID: "e9eb58fe-4a3a-4b5b-b050-583b1948236b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:23:31.572882 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.572857 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9eb58fe-4a3a-4b5b-b050-583b1948236b-kube-api-access-bh9hj" (OuterVolumeSpecName: "kube-api-access-bh9hj") pod "e9eb58fe-4a3a-4b5b-b050-583b1948236b" (UID: "e9eb58fe-4a3a-4b5b-b050-583b1948236b"). InnerVolumeSpecName "kube-api-access-bh9hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:23:31.672289 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.672226 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9eb58fe-4a3a-4b5b-b050-583b1948236b-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:23:31.672289 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.672251 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh9hj\" (UniqueName: \"kubernetes.io/projected/e9eb58fe-4a3a-4b5b-b050-583b1948236b-kube-api-access-bh9hj\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:23:31.746756 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.746731 2564 generic.go:356] "Generic (PLEG): container finished" podID="e9eb58fe-4a3a-4b5b-b050-583b1948236b" containerID="f0329219d6ae19f7c14ecaed764f8cdc803d1e1311f91a99ee38bf1c5ac47994" exitCode=0 Sep 17 18:23:31.746848 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.746775 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" Sep 17 18:23:31.746848 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.746818 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" event={"ID":"e9eb58fe-4a3a-4b5b-b050-583b1948236b","Type":"ContainerDied","Data":"f0329219d6ae19f7c14ecaed764f8cdc803d1e1311f91a99ee38bf1c5ac47994"} Sep 17 18:23:31.746938 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.746869 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr" event={"ID":"e9eb58fe-4a3a-4b5b-b050-583b1948236b","Type":"ContainerDied","Data":"665b6ec3049464b8a0c15a86ba479851dea7f4c393f6b4ecd809b6c9d26d010e"} Sep 17 18:23:31.746938 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.746893 2564 scope.go:117] "RemoveContainer" containerID="f0329219d6ae19f7c14ecaed764f8cdc803d1e1311f91a99ee38bf1c5ac47994" Sep 17 18:23:31.756782 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.756765 2564 scope.go:117] "RemoveContainer" containerID="f0329219d6ae19f7c14ecaed764f8cdc803d1e1311f91a99ee38bf1c5ac47994" Sep 17 18:23:31.757075 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:23:31.757059 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0329219d6ae19f7c14ecaed764f8cdc803d1e1311f91a99ee38bf1c5ac47994\": container with ID starting with f0329219d6ae19f7c14ecaed764f8cdc803d1e1311f91a99ee38bf1c5ac47994 not found: ID does not exist" containerID="f0329219d6ae19f7c14ecaed764f8cdc803d1e1311f91a99ee38bf1c5ac47994" Sep 17 18:23:31.757122 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.757086 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0329219d6ae19f7c14ecaed764f8cdc803d1e1311f91a99ee38bf1c5ac47994"} err="failed to get container status \"f0329219d6ae19f7c14ecaed764f8cdc803d1e1311f91a99ee38bf1c5ac47994\": rpc error: code = NotFound desc = could not find container \"f0329219d6ae19f7c14ecaed764f8cdc803d1e1311f91a99ee38bf1c5ac47994\": container with ID starting with f0329219d6ae19f7c14ecaed764f8cdc803d1e1311f91a99ee38bf1c5ac47994 not found: ID does not exist" Sep 17 18:23:31.768658 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.768635 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr"] Sep 17 18:23:31.772082 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:31.772059 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99ac10-67f9d6f7fd-7p7dr"] Sep 17 18:23:33.452505 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:33.452473 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9eb58fe-4a3a-4b5b-b050-583b1948236b" path="/var/lib/kubelet/pods/e9eb58fe-4a3a-4b5b-b050-583b1948236b/volumes" Sep 17 18:23:55.592865 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:23:55.592797 2564 scope.go:117] "RemoveContainer" containerID="24614bbc30e806dc11d15ddda92560ffd352b7e475664d8975109e4ba471bc7a" Sep 17 18:24:14.627467 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:14.627437 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh"] Sep 17 18:24:14.627926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:14.627693 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" podUID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerName="istio-proxy" containerID="cri-o://97a3fcfa913094d91c57fea652f5fe93fd5193123d0e9f33855771ffd4c23730" gracePeriod=30 Sep 17 18:24:19.887581 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.887556 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:24:19.994576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.994508 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-envoy\") pod \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " Sep 17 18:24:19.994576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.994560 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-workload-certs\") pod \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " Sep 17 18:24:19.994778 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.994586 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-workload-socket\") pod \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " Sep 17 18:24:19.994778 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.994601 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-credential-socket\") pod \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " Sep 17 18:24:19.994778 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.994643 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istiod-ca-cert\") pod \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " Sep 17 18:24:19.994778 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.994710 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-token\") pod \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " Sep 17 18:24:19.994778 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.994757 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-podinfo\") pod \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " Sep 17 18:24:19.995040 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.994791 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-data\") pod \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " Sep 17 18:24:19.995040 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.994836 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25j4g\" (UniqueName: \"kubernetes.io/projected/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-kube-api-access-25j4g\") pod \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\" (UID: \"1d4f7a4f-0dbe-424b-93ce-0215bd26e252\") " Sep 17 18:24:19.995040 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.994907 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1d4f7a4f-0dbe-424b-93ce-0215bd26e252" (UID: "1d4f7a4f-0dbe-424b-93ce-0215bd26e252"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:24:19.995040 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.994927 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1d4f7a4f-0dbe-424b-93ce-0215bd26e252" (UID: "1d4f7a4f-0dbe-424b-93ce-0215bd26e252"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:24:19.995040 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.994906 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1d4f7a4f-0dbe-424b-93ce-0215bd26e252" (UID: "1d4f7a4f-0dbe-424b-93ce-0215bd26e252"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:24:19.995289 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.995124 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-data" (OuterVolumeSpecName: "istio-data") pod "1d4f7a4f-0dbe-424b-93ce-0215bd26e252" (UID: "1d4f7a4f-0dbe-424b-93ce-0215bd26e252"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:24:19.995289 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.995228 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:24:19.995289 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.995245 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:24:19.995289 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.995260 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:24:19.995289 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.995275 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:24:19.995618 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.995590 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1d4f7a4f-0dbe-424b-93ce-0215bd26e252" (UID: "1d4f7a4f-0dbe-424b-93ce-0215bd26e252"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:24:19.996918 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.996892 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1d4f7a4f-0dbe-424b-93ce-0215bd26e252" (UID: "1d4f7a4f-0dbe-424b-93ce-0215bd26e252"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:24:19.997031 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.996988 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-token" (OuterVolumeSpecName: "istio-token") pod "1d4f7a4f-0dbe-424b-93ce-0215bd26e252" (UID: "1d4f7a4f-0dbe-424b-93ce-0215bd26e252"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:24:19.997560 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.997503 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-kube-api-access-25j4g" (OuterVolumeSpecName: "kube-api-access-25j4g") pod "1d4f7a4f-0dbe-424b-93ce-0215bd26e252" (UID: "1d4f7a4f-0dbe-424b-93ce-0215bd26e252"). InnerVolumeSpecName "kube-api-access-25j4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:24:19.997650 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:19.997562 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1d4f7a4f-0dbe-424b-93ce-0215bd26e252" (UID: "1d4f7a4f-0dbe-424b-93ce-0215bd26e252"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:24:20.044849 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.044817 2564 generic.go:356] "Generic (PLEG): container finished" podID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerID="97a3fcfa913094d91c57fea652f5fe93fd5193123d0e9f33855771ffd4c23730" exitCode=0 Sep 17 18:24:20.044937 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.044869 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" Sep 17 18:24:20.044937 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.044915 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" event={"ID":"1d4f7a4f-0dbe-424b-93ce-0215bd26e252","Type":"ContainerDied","Data":"97a3fcfa913094d91c57fea652f5fe93fd5193123d0e9f33855771ffd4c23730"} Sep 17 18:24:20.045013 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.044971 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh" event={"ID":"1d4f7a4f-0dbe-424b-93ce-0215bd26e252","Type":"ContainerDied","Data":"3f78d69df62f44b801bd14189dabdb8e52971da3956e600e58892dd3a4a51a8c"} Sep 17 18:24:20.045013 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.044995 2564 scope.go:117] "RemoveContainer" containerID="97a3fcfa913094d91c57fea652f5fe93fd5193123d0e9f33855771ffd4c23730" Sep 17 18:24:20.055096 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.055076 2564 scope.go:117] "RemoveContainer" containerID="490529dc704ee80d3eb758868f2eedb4e1e45df0280253827c4ccf134276e3c1" Sep 17 18:24:20.063722 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.063707 2564 scope.go:117] "RemoveContainer" containerID="97a3fcfa913094d91c57fea652f5fe93fd5193123d0e9f33855771ffd4c23730" Sep 17 18:24:20.063960 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:24:20.063942 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a3fcfa913094d91c57fea652f5fe93fd5193123d0e9f33855771ffd4c23730\": container with ID starting with 97a3fcfa913094d91c57fea652f5fe93fd5193123d0e9f33855771ffd4c23730 not found: ID does not exist" containerID="97a3fcfa913094d91c57fea652f5fe93fd5193123d0e9f33855771ffd4c23730" Sep 17 18:24:20.064076 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.063964 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a3fcfa913094d91c57fea652f5fe93fd5193123d0e9f33855771ffd4c23730"} err="failed to get container status \"97a3fcfa913094d91c57fea652f5fe93fd5193123d0e9f33855771ffd4c23730\": rpc error: code = NotFound desc = could not find container \"97a3fcfa913094d91c57fea652f5fe93fd5193123d0e9f33855771ffd4c23730\": container with ID starting with 97a3fcfa913094d91c57fea652f5fe93fd5193123d0e9f33855771ffd4c23730 not found: ID does not exist" Sep 17 18:24:20.064076 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.063978 2564 scope.go:117] "RemoveContainer" containerID="490529dc704ee80d3eb758868f2eedb4e1e45df0280253827c4ccf134276e3c1" Sep 17 18:24:20.064242 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:24:20.064219 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490529dc704ee80d3eb758868f2eedb4e1e45df0280253827c4ccf134276e3c1\": container with ID starting with 490529dc704ee80d3eb758868f2eedb4e1e45df0280253827c4ccf134276e3c1 not found: ID does not exist" containerID="490529dc704ee80d3eb758868f2eedb4e1e45df0280253827c4ccf134276e3c1" Sep 17 18:24:20.064286 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.064248 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490529dc704ee80d3eb758868f2eedb4e1e45df0280253827c4ccf134276e3c1"} err="failed to get container status \"490529dc704ee80d3eb758868f2eedb4e1e45df0280253827c4ccf134276e3c1\": rpc error: code = NotFound desc = could not find container \"490529dc704ee80d3eb758868f2eedb4e1e45df0280253827c4ccf134276e3c1\": container with ID starting with 490529dc704ee80d3eb758868f2eedb4e1e45df0280253827c4ccf134276e3c1 not found: ID does not exist" Sep 17 18:24:20.068796 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.068775 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh"] Sep 17 18:24:20.075038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.075018 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-99ac10-openshift-default-cdc8788b6-fcvwh"] Sep 17 18:24:20.095845 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.095821 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:24:20.095845 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.095841 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:24:20.095961 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.095850 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25j4g\" (UniqueName: \"kubernetes.io/projected/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-kube-api-access-25j4g\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:24:20.095961 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.095859 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:24:20.095961 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:20.095868 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4f7a4f-0dbe-424b-93ce-0215bd26e252-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:24:21.451733 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:21.451697 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" path="/var/lib/kubelet/pods/1d4f7a4f-0dbe-424b-93ce-0215bd26e252/volumes" Sep 17 18:24:23.409898 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.409864 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm"] Sep 17 18:24:23.410426 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.410394 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerName="istio-proxy" Sep 17 18:24:23.410426 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.410427 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerName="istio-proxy" Sep 17 18:24:23.410627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.410450 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerName="istio-proxy" Sep 17 18:24:23.410627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.410459 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerName="istio-proxy" Sep 17 18:24:23.410627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.410477 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerName="istio-proxy" Sep 17 18:24:23.410627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.410484 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerName="istio-proxy" Sep 17 18:24:23.410627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.410494 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerName="istio-proxy" Sep 17 18:24:23.410627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.410499 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerName="istio-proxy" Sep 17 18:24:23.410627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.410514 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9eb58fe-4a3a-4b5b-b050-583b1948236b" containerName="payload-pre-processing" Sep 17 18:24:23.410627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.410519 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9eb58fe-4a3a-4b5b-b050-583b1948236b" containerName="payload-pre-processing" Sep 17 18:24:23.410627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.410627 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerName="istio-proxy" Sep 17 18:24:23.410946 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.410641 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerName="istio-proxy" Sep 17 18:24:23.410946 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.410649 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9eb58fe-4a3a-4b5b-b050-583b1948236b" containerName="payload-pre-processing" Sep 17 18:24:23.415416 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.415398 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" Sep 17 18:24:23.423386 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.423358 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-95a92b\"" Sep 17 18:24:23.423726 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.423704 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-95a92b-dockercfg-9bzj2\"" Sep 17 18:24:23.427377 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.427353 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc"] Sep 17 18:24:23.428058 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.428038 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d4f7a4f-0dbe-424b-93ce-0215bd26e252" containerName="istio-proxy" Sep 17 18:24:23.428117 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.428063 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e8655d-b888-4cf2-b25a-e1b1eb9ab8de" containerName="istio-proxy" Sep 17 18:24:23.432958 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.432940 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm"] Sep 17 18:24:23.433046 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.433016 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" Sep 17 18:24:23.440065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.440044 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc"] Sep 17 18:24:23.529359 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.529317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2phl8\" (UniqueName: \"kubernetes.io/projected/16834e97-f84f-48af-b796-607dcef66fb4-kube-api-access-2phl8\") pod \"payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm\" (UID: \"16834e97-f84f-48af-b796-607dcef66fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" Sep 17 18:24:23.529587 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.529379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8lvq\" (UniqueName: \"kubernetes.io/projected/9c8d7e85-838b-4210-b470-2d1972e63bf6-kube-api-access-f8lvq\") pod \"payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc\" (UID: \"9c8d7e85-838b-4210-b470-2d1972e63bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" Sep 17 18:24:23.529587 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.529575 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8d7e85-838b-4210-b470-2d1972e63bf6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc\" (UID: \"9c8d7e85-838b-4210-b470-2d1972e63bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" Sep 17 18:24:23.529731 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.529643 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16834e97-f84f-48af-b796-607dcef66fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm\" (UID: \"16834e97-f84f-48af-b796-607dcef66fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" Sep 17 18:24:23.630833 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.630798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16834e97-f84f-48af-b796-607dcef66fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm\" (UID: \"16834e97-f84f-48af-b796-607dcef66fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" Sep 17 18:24:23.630997 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.630852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2phl8\" (UniqueName: \"kubernetes.io/projected/16834e97-f84f-48af-b796-607dcef66fb4-kube-api-access-2phl8\") pod \"payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm\" (UID: \"16834e97-f84f-48af-b796-607dcef66fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" Sep 17 18:24:23.630997 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.630877 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8lvq\" (UniqueName: \"kubernetes.io/projected/9c8d7e85-838b-4210-b470-2d1972e63bf6-kube-api-access-f8lvq\") pod \"payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc\" (UID: \"9c8d7e85-838b-4210-b470-2d1972e63bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" Sep 17 18:24:23.631115 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.631014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8d7e85-838b-4210-b470-2d1972e63bf6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc\" (UID: \"9c8d7e85-838b-4210-b470-2d1972e63bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" Sep 17 18:24:23.631603 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.631583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16834e97-f84f-48af-b796-607dcef66fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm\" (UID: \"16834e97-f84f-48af-b796-607dcef66fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" Sep 17 18:24:23.631763 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.631747 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8d7e85-838b-4210-b470-2d1972e63bf6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc\" (UID: \"9c8d7e85-838b-4210-b470-2d1972e63bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" Sep 17 18:24:23.639996 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.639976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8lvq\" (UniqueName: \"kubernetes.io/projected/9c8d7e85-838b-4210-b470-2d1972e63bf6-kube-api-access-f8lvq\") pod \"payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc\" (UID: \"9c8d7e85-838b-4210-b470-2d1972e63bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" Sep 17 18:24:23.640197 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.640177 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2phl8\" (UniqueName: \"kubernetes.io/projected/16834e97-f84f-48af-b796-607dcef66fb4-kube-api-access-2phl8\") pod \"payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm\" (UID: \"16834e97-f84f-48af-b796-607dcef66fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" Sep 17 18:24:23.736820 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.736758 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" Sep 17 18:24:23.747294 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:23.747274 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" Sep 17 18:24:24.016399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:24.016371 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm"] Sep 17 18:24:24.018570 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:24:24.018547 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16834e97_f84f_48af_b796_607dcef66fb4.slice/crio-aacb2a34db6b652b1429b39d905ffeb1e9e28ac3e2c871c62481847c32369233 WatchSource:0}: Error finding container aacb2a34db6b652b1429b39d905ffeb1e9e28ac3e2c871c62481847c32369233: Status 404 returned error can't find the container with id aacb2a34db6b652b1429b39d905ffeb1e9e28ac3e2c871c62481847c32369233 Sep 17 18:24:24.032824 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:24:24.032796 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8d7e85_838b_4210_b470_2d1972e63bf6.slice/crio-caf145813e071b319f089e04afd3ec1819e5664c9b47208e785bb8f5cac60e7b WatchSource:0}: Error finding container caf145813e071b319f089e04afd3ec1819e5664c9b47208e785bb8f5cac60e7b: Status 404 returned error can't find the container with id caf145813e071b319f089e04afd3ec1819e5664c9b47208e785bb8f5cac60e7b Sep 17 18:24:24.033579 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:24.033551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc"] Sep 17 18:24:24.073597 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:24.073568 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" event={"ID":"9c8d7e85-838b-4210-b470-2d1972e63bf6","Type":"ContainerStarted","Data":"caf145813e071b319f089e04afd3ec1819e5664c9b47208e785bb8f5cac60e7b"} Sep 17 18:24:24.075292 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:24.075270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" event={"ID":"16834e97-f84f-48af-b796-607dcef66fb4","Type":"ContainerStarted","Data":"aacb2a34db6b652b1429b39d905ffeb1e9e28ac3e2c871c62481847c32369233"} Sep 17 18:24:25.083144 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:25.083106 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" event={"ID":"9c8d7e85-838b-4210-b470-2d1972e63bf6","Type":"ContainerStarted","Data":"34cde97c73c3392ce9dc6b82b9caf3d6fbe57c30ed93b35a399c312f08b95f16"} Sep 17 18:24:25.083570 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:25.083176 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" Sep 17 18:24:25.084834 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:25.084808 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" event={"ID":"16834e97-f84f-48af-b796-607dcef66fb4","Type":"ContainerStarted","Data":"f06fcfdfba05af0f8471e73e03ead05a435653d9090c1b8003995493cf11201b"} Sep 17 18:24:25.084935 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:25.084860 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" Sep 17 18:24:25.142013 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:25.141964 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" podStartSLOduration=2.141946389 podStartE2EDuration="2.141946389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:25.111193227 +0000 UTC m=+1290.121380364" watchObservedRunningTime="2026-09-17 18:24:25.141946389 +0000 UTC m=+1290.152133526" Sep 17 18:24:25.144469 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:25.144433 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" podStartSLOduration=2.1444191200000002 podStartE2EDuration="2.14441912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:25.14177229 +0000 UTC m=+1290.151959440" watchObservedRunningTime="2026-09-17 18:24:25.14441912 +0000 UTC m=+1290.154606259" Sep 17 18:24:36.092559 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:36.092505 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" Sep 17 18:24:36.093054 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:36.092592 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" Sep 17 18:24:48.972554 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:48.972489 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s"] Sep 17 18:24:48.977763 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:48.977736 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:48.980474 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:48.980447 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d81d7c-openshift-default-dockercfg-j7l69\"" Sep 17 18:24:48.989634 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:48.989605 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s"] Sep 17 18:24:49.049754 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.049722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvxck\" (UniqueName: \"kubernetes.io/projected/efdef7ba-ba6b-464b-88d6-618baeab04bc-kube-api-access-wvxck\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.049865 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.049777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-podinfo\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.049865 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.049812 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-credential-socket\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.049865 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.049856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-token\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.049989 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.049967 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-workload-certs\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.050027 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.050002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efdef7ba-ba6b-464b-88d6-618baeab04bc-istiod-ca-cert\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.050079 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.050068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-workload-socket\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.050115 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.050087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-envoy\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.050149 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.050111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-data\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.151092 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.151060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-workload-certs\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.151269 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.151100 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efdef7ba-ba6b-464b-88d6-618baeab04bc-istiod-ca-cert\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.151269 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.151142 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-workload-socket\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.151269 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.151159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-envoy\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.151269 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.151183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-data\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.151269 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.151218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxck\" (UniqueName: \"kubernetes.io/projected/efdef7ba-ba6b-464b-88d6-618baeab04bc-kube-api-access-wvxck\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.151269 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.151242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-podinfo\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.151269 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.151258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-credential-socket\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.151703 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.151290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-token\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.151703 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.151493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-workload-certs\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.151703 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.151608 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-workload-socket\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.151881 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.151807 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-data\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.152004 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.151980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efdef7ba-ba6b-464b-88d6-618baeab04bc-istiod-ca-cert\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.152259 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.152237 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-credential-socket\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.153717 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.153696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-envoy\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.154033 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.154013 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-podinfo\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.158927 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.158903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-token\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.159293 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.159270 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxck\" (UniqueName: \"kubernetes.io/projected/efdef7ba-ba6b-464b-88d6-618baeab04bc-kube-api-access-wvxck\") pod \"e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.290014 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.289931 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:49.573053 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.572243 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s"] Sep 17 18:24:49.575723 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:24:49.575697 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdef7ba_ba6b_464b_88d6_618baeab04bc.slice/crio-0700ac560b1bc5ddeeb3f962eea1191039a241dc38bf58bb4f6cd86af0018ba5 WatchSource:0}: Error finding container 0700ac560b1bc5ddeeb3f962eea1191039a241dc38bf58bb4f6cd86af0018ba5: Status 404 returned error can't find the container with id 0700ac560b1bc5ddeeb3f962eea1191039a241dc38bf58bb4f6cd86af0018ba5 Sep 17 18:24:49.578274 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.578229 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:24:49.578387 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.578315 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:24:49.578455 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:49.578386 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:24:50.246713 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:50.246672 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" event={"ID":"efdef7ba-ba6b-464b-88d6-618baeab04bc","Type":"ContainerStarted","Data":"b9fbf131888229e4e6f6b4050901320b06816a03fe3e1ca60b5cf3092e17a21a"} Sep 17 18:24:50.247068 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:50.246733 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" event={"ID":"efdef7ba-ba6b-464b-88d6-618baeab04bc","Type":"ContainerStarted","Data":"0700ac560b1bc5ddeeb3f962eea1191039a241dc38bf58bb4f6cd86af0018ba5"} Sep 17 18:24:50.269601 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:50.269551 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" podStartSLOduration=2.269517551 podStartE2EDuration="2.269517551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:50.267405649 +0000 UTC m=+1315.277592784" watchObservedRunningTime="2026-09-17 18:24:50.269517551 +0000 UTC m=+1315.279704687" Sep 17 18:24:50.290585 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:50.290518 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:50.295598 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:50.295577 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:51.253002 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:51.252973 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:51.253719 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:51.253698 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:24:55.622730 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:55.622699 2564 scope.go:117] "RemoveContainer" containerID="0b65e41c034a9df3c35ee66cb65e5df833a2651811e89c045918379f52d87f98" Sep 17 18:24:55.631505 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:55.631482 2564 scope.go:117] "RemoveContainer" containerID="131b271b2227ba7decdb17ee83e0f380c3a31d47b74416cb7b3190c1fc0de8fe" Sep 17 18:24:58.094447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:58.094411 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv"] Sep 17 18:24:58.099912 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:58.099890 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" Sep 17 18:24:58.102230 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:58.102209 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d81d7c\"" Sep 17 18:24:58.102336 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:58.102276 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d81d7c-dockercfg-q872h\"" Sep 17 18:24:58.111151 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:58.111124 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv"] Sep 17 18:24:58.132639 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:58.132616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589dcc22-437e-4bbe-964d-a943c6d7c9ec-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv\" (UID: \"589dcc22-437e-4bbe-964d-a943c6d7c9ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" Sep 17 18:24:58.132738 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:58.132661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2dgr\" (UniqueName: \"kubernetes.io/projected/589dcc22-437e-4bbe-964d-a943c6d7c9ec-kube-api-access-l2dgr\") pod \"payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv\" (UID: \"589dcc22-437e-4bbe-964d-a943c6d7c9ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" Sep 17 18:24:58.233914 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:58.233883 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589dcc22-437e-4bbe-964d-a943c6d7c9ec-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv\" (UID: \"589dcc22-437e-4bbe-964d-a943c6d7c9ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" Sep 17 18:24:58.234096 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:58.233935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2dgr\" (UniqueName: \"kubernetes.io/projected/589dcc22-437e-4bbe-964d-a943c6d7c9ec-kube-api-access-l2dgr\") pod \"payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv\" (UID: \"589dcc22-437e-4bbe-964d-a943c6d7c9ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" Sep 17 18:24:58.234590 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:58.234571 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589dcc22-437e-4bbe-964d-a943c6d7c9ec-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv\" (UID: \"589dcc22-437e-4bbe-964d-a943c6d7c9ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" Sep 17 18:24:58.242201 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:58.242181 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2dgr\" (UniqueName: \"kubernetes.io/projected/589dcc22-437e-4bbe-964d-a943c6d7c9ec-kube-api-access-l2dgr\") pod \"payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv\" (UID: \"589dcc22-437e-4bbe-964d-a943c6d7c9ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" Sep 17 18:24:58.412217 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:58.412154 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" Sep 17 18:24:58.686749 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:58.686724 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv"] Sep 17 18:24:58.687589 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:24:58.687561 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589dcc22_437e_4bbe_964d_a943c6d7c9ec.slice/crio-0ec5433c3b54e5a80d7ef18458ec827bcdcd44a0cf37f7c0633b4ccb9a4c4217 WatchSource:0}: Error finding container 0ec5433c3b54e5a80d7ef18458ec827bcdcd44a0cf37f7c0633b4ccb9a4c4217: Status 404 returned error can't find the container with id 0ec5433c3b54e5a80d7ef18458ec827bcdcd44a0cf37f7c0633b4ccb9a4c4217 Sep 17 18:24:59.323016 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:59.322981 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" event={"ID":"589dcc22-437e-4bbe-964d-a943c6d7c9ec","Type":"ContainerStarted","Data":"bfdeb41e9141bb6c7613ccc1ba1276c9088a84300d4ba99bf714d2942b9b750e"} Sep 17 18:24:59.323458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:59.323038 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" event={"ID":"589dcc22-437e-4bbe-964d-a943c6d7c9ec","Type":"ContainerStarted","Data":"0ec5433c3b54e5a80d7ef18458ec827bcdcd44a0cf37f7c0633b4ccb9a4c4217"} Sep 17 18:24:59.323458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:59.323151 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" Sep 17 18:24:59.342227 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:24:59.342181 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" podStartSLOduration=1.342166636 podStartE2EDuration="1.342166636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:59.340616112 +0000 UTC m=+1324.350803248" watchObservedRunningTime="2026-09-17 18:24:59.342166636 +0000 UTC m=+1324.352353771" Sep 17 18:25:10.330685 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:10.330652 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" Sep 17 18:25:22.294684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:22.294642 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 17 18:25:22.295116 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:22.294718 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 18:25:22.491765 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:22.491735 2564 generic.go:356] "Generic (PLEG): container finished" podID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerID="b9fbf131888229e4e6f6b4050901320b06816a03fe3e1ca60b5cf3092e17a21a" exitCode=0 Sep 17 18:25:22.491893 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:22.491769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" event={"ID":"efdef7ba-ba6b-464b-88d6-618baeab04bc","Type":"ContainerDied","Data":"b9fbf131888229e4e6f6b4050901320b06816a03fe3e1ca60b5cf3092e17a21a"} Sep 17 18:25:22.492176 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:22.492136 2564 scope.go:117] "RemoveContainer" containerID="b9fbf131888229e4e6f6b4050901320b06816a03fe3e1ca60b5cf3092e17a21a" Sep 17 18:25:22.493328 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:22.493298 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:25:22.493383 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:22.493368 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:25:22.493426 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:22.493400 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:25:23.499432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:23.499398 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" event={"ID":"efdef7ba-ba6b-464b-88d6-618baeab04bc","Type":"ContainerStarted","Data":"bf088340b337ad9a56b4f5a2e663fa637b5f36ef7b6f0a4c94ebce48c86f12c9"} Sep 17 18:25:24.290710 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:24.290667 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:25:24.291014 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:24.290732 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 17 18:25:24.291170 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:24.291146 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 17 18:25:25.290747 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:25.290714 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 17 18:25:25.291218 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:25.290772 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 17 18:25:26.290547 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:26.290499 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 17 18:25:26.290727 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:26.290590 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 17 18:25:27.291204 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:27.291168 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 17 18:25:27.291617 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:27.291256 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 17 18:25:28.291077 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:28.291032 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 17 18:25:28.291259 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:28.291128 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 17 18:25:29.290923 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:29.290885 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 17 18:25:29.291130 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:29.290948 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 17 18:25:30.293744 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:30.293710 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:25:30.294105 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:30.294031 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:25:30.294724 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:25:30.294708 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:27:55.677518 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:27:55.677487 2564 scope.go:117] "RemoveContainer" containerID="f0f6be8e25c624c381df626c03f58e6aa18d9e096a22b04fb9c4eddb484068ea" Sep 17 18:29:03.442127 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.442093 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb"] Sep 17 18:29:03.446108 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.446084 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.449060 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.449033 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b06908-openshift-default-dockercfg-pmlx5\"" Sep 17 18:29:03.459184 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.459158 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb"] Sep 17 18:29:03.587877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.587839 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5c9t\" (UniqueName: \"kubernetes.io/projected/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-kube-api-access-z5c9t\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.588170 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.588146 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-credential-socket\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.588740 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.588714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istiod-ca-cert\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.588869 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.588790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-data\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.588869 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.588817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-token\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.588976 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.588965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-workload-socket\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.589030 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.589001 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-envoy\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.589081 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.589051 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-podinfo\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.589147 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.589126 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-workload-certs\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.696079 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.695987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-envoy\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.696079 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.696030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-podinfo\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.696079 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.696061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-workload-certs\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.696392 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.696094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5c9t\" (UniqueName: \"kubernetes.io/projected/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-kube-api-access-z5c9t\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.696392 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.696115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-credential-socket\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.696392 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.696240 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istiod-ca-cert\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.696392 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.696321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-data\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.696392 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.696348 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-token\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.696699 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.696460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-workload-socket\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.696699 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.696629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-credential-socket\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.696982 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.696727 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-data\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.696982 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.696834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-workload-certs\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.697094 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.697024 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istiod-ca-cert\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.697153 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.697087 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-workload-socket\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.698618 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.698588 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-envoy\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.698846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.698826 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-podinfo\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.708106 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.708079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-token\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.708472 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.708455 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5c9t\" (UniqueName: \"kubernetes.io/projected/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-kube-api-access-z5c9t\") pod \"e2e-api-b-b06908-openshift-default-bd57dd984-6wddb\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:03.760057 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:03.760029 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:04.054640 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:04.054608 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb"] Sep 17 18:29:04.055155 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:29:04.055133 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21c0cc6_1c7b_4d20_8fa1_2f01224de923.slice/crio-42902be1acdfb6ca8eed153875c5e1d3edc963d481f088a2862b00567be86dc2 WatchSource:0}: Error finding container 42902be1acdfb6ca8eed153875c5e1d3edc963d481f088a2862b00567be86dc2: Status 404 returned error can't find the container with id 42902be1acdfb6ca8eed153875c5e1d3edc963d481f088a2862b00567be86dc2 Sep 17 18:29:04.057586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:04.057553 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:29:04.057659 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:04.057624 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:29:04.057659 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:04.057653 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:29:04.884126 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:04.884089 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" event={"ID":"e21c0cc6-1c7b-4d20-8fa1-2f01224de923","Type":"ContainerStarted","Data":"6197b2d4f63aa2cedbbfef38a88669555eddea014605a546477d7d7aa025e5fb"} Sep 17 18:29:04.884561 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:04.884148 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" event={"ID":"e21c0cc6-1c7b-4d20-8fa1-2f01224de923","Type":"ContainerStarted","Data":"42902be1acdfb6ca8eed153875c5e1d3edc963d481f088a2862b00567be86dc2"} Sep 17 18:29:04.909043 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:04.908999 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" podStartSLOduration=1.9089845890000001 podStartE2EDuration="1.908984589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:04.905164915 +0000 UTC m=+1569.915352042" watchObservedRunningTime="2026-09-17 18:29:04.908984589 +0000 UTC m=+1569.919171724" Sep 17 18:29:05.760332 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:05.760306 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:05.766136 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:05.766114 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:05.890966 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:05.890939 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:05.891951 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:05.891927 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:13.251385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.251346 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx"] Sep 17 18:29:13.255363 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.255343 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" Sep 17 18:29:13.258457 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.258436 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b06908-dockercfg-9zplr\"" Sep 17 18:29:13.259497 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.259484 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b06908\"" Sep 17 18:29:13.276006 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.275981 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx"] Sep 17 18:29:13.288080 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.288055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2msv\" (UniqueName: \"kubernetes.io/projected/86d6d36a-bc35-4381-a8bd-9badc7e3d03a-kube-api-access-m2msv\") pod \"payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx\" (UID: \"86d6d36a-bc35-4381-a8bd-9badc7e3d03a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" Sep 17 18:29:13.288176 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.288081 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d6d36a-bc35-4381-a8bd-9badc7e3d03a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx\" (UID: \"86d6d36a-bc35-4381-a8bd-9badc7e3d03a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" Sep 17 18:29:13.388835 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.388808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2msv\" (UniqueName: \"kubernetes.io/projected/86d6d36a-bc35-4381-a8bd-9badc7e3d03a-kube-api-access-m2msv\") pod \"payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx\" (UID: \"86d6d36a-bc35-4381-a8bd-9badc7e3d03a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" Sep 17 18:29:13.388947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.388842 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d6d36a-bc35-4381-a8bd-9badc7e3d03a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx\" (UID: \"86d6d36a-bc35-4381-a8bd-9badc7e3d03a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" Sep 17 18:29:13.389551 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.389507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d6d36a-bc35-4381-a8bd-9badc7e3d03a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx\" (UID: \"86d6d36a-bc35-4381-a8bd-9badc7e3d03a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" Sep 17 18:29:13.397317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.397298 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2msv\" (UniqueName: \"kubernetes.io/projected/86d6d36a-bc35-4381-a8bd-9badc7e3d03a-kube-api-access-m2msv\") pod \"payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx\" (UID: \"86d6d36a-bc35-4381-a8bd-9badc7e3d03a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" Sep 17 18:29:13.566380 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.566360 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" Sep 17 18:29:13.844999 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.844971 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx"] Sep 17 18:29:13.846796 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:29:13.846768 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d6d36a_bc35_4381_a8bd_9badc7e3d03a.slice/crio-7f5f52d416e2bd9bbe3716a29a2c3ddd538ec5afbbc07615a393dbc2bebb73b5 WatchSource:0}: Error finding container 7f5f52d416e2bd9bbe3716a29a2c3ddd538ec5afbbc07615a393dbc2bebb73b5: Status 404 returned error can't find the container with id 7f5f52d416e2bd9bbe3716a29a2c3ddd538ec5afbbc07615a393dbc2bebb73b5 Sep 17 18:29:13.944871 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.944841 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" event={"ID":"86d6d36a-bc35-4381-a8bd-9badc7e3d03a","Type":"ContainerStarted","Data":"f73a853b7f5495602727bc132a8aa69a830d51636f8ee1b47c70e7cdeb450fe9"} Sep 17 18:29:13.944976 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.944893 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" event={"ID":"86d6d36a-bc35-4381-a8bd-9badc7e3d03a","Type":"ContainerStarted","Data":"7f5f52d416e2bd9bbe3716a29a2c3ddd538ec5afbbc07615a393dbc2bebb73b5"} Sep 17 18:29:13.944976 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.944923 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" Sep 17 18:29:13.961836 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:13.961772 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" podStartSLOduration=0.961751167 podStartE2EDuration="961.751167ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:13.961153407 +0000 UTC m=+1578.971340548" watchObservedRunningTime="2026-09-17 18:29:13.961751167 +0000 UTC m=+1578.971938307" Sep 17 18:29:24.953704 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:24.953671 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" Sep 17 18:29:38.089230 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:38.089196 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7"] Sep 17 18:29:38.093097 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:38.093079 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7" Sep 17 18:29:38.095635 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:38.095614 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\"" Sep 17 18:29:38.095795 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:38.095717 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 18:29:38.104496 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:38.104469 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7"] Sep 17 18:29:38.217746 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:38.217714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/607a95a0-8f20-474a-b841-385ca1d957c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7\" (UID: \"607a95a0-8f20-474a-b841-385ca1d957c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7" Sep 17 18:29:38.319099 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:38.319063 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/607a95a0-8f20-474a-b841-385ca1d957c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7\" (UID: \"607a95a0-8f20-474a-b841-385ca1d957c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7" Sep 17 18:29:38.319771 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:38.319750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/607a95a0-8f20-474a-b841-385ca1d957c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7\" (UID: \"607a95a0-8f20-474a-b841-385ca1d957c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7" Sep 17 18:29:38.407416 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:38.407339 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7" Sep 17 18:29:38.707623 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:38.707596 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7"] Sep 17 18:29:38.709507 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:29:38.709481 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607a95a0_8f20_474a_b841_385ca1d957c2.slice/crio-a75fc0d3db278c93de4f5f596b83d5e794264698e21c34c6928f37a7a2c31f89 WatchSource:0}: Error finding container a75fc0d3db278c93de4f5f596b83d5e794264698e21c34c6928f37a7a2c31f89: Status 404 returned error can't find the container with id a75fc0d3db278c93de4f5f596b83d5e794264698e21c34c6928f37a7a2c31f89 Sep 17 18:29:39.111222 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:39.111186 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7" event={"ID":"607a95a0-8f20-474a-b841-385ca1d957c2","Type":"ContainerStarted","Data":"a75fc0d3db278c93de4f5f596b83d5e794264698e21c34c6928f37a7a2c31f89"} Sep 17 18:29:40.120100 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:40.120067 2564 generic.go:356] "Generic (PLEG): container finished" podID="607a95a0-8f20-474a-b841-385ca1d957c2" containerID="453015822ddb127225ddd3f1904ea6a1a9d4a656ff02683ee31ebab73996605c" exitCode=0 Sep 17 18:29:40.120607 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:40.120134 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7" event={"ID":"607a95a0-8f20-474a-b841-385ca1d957c2","Type":"ContainerDied","Data":"453015822ddb127225ddd3f1904ea6a1a9d4a656ff02683ee31ebab73996605c"} Sep 17 18:29:41.269990 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:41.269964 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7" Sep 17 18:29:41.352080 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:41.352048 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/607a95a0-8f20-474a-b841-385ca1d957c2-maas-api-service-ca\") pod \"607a95a0-8f20-474a-b841-385ca1d957c2\" (UID: \"607a95a0-8f20-474a-b841-385ca1d957c2\") " Sep 17 18:29:41.352454 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:41.352435 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/607a95a0-8f20-474a-b841-385ca1d957c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "607a95a0-8f20-474a-b841-385ca1d957c2" (UID: "607a95a0-8f20-474a-b841-385ca1d957c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:29:41.453327 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:41.453248 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/607a95a0-8f20-474a-b841-385ca1d957c2-maas-api-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:29:42.136136 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:42.136100 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7" Sep 17 18:29:42.136136 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:42.136118 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7" event={"ID":"607a95a0-8f20-474a-b841-385ca1d957c2","Type":"ContainerDied","Data":"a75fc0d3db278c93de4f5f596b83d5e794264698e21c34c6928f37a7a2c31f89"} Sep 17 18:29:42.136692 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:42.136173 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a75fc0d3db278c93de4f5f596b83d5e794264698e21c34c6928f37a7a2c31f89" Sep 17 18:29:43.144014 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:43.143985 2564 generic.go:356] "Generic (PLEG): container finished" podID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerID="6197b2d4f63aa2cedbbfef38a88669555eddea014605a546477d7d7aa025e5fb" exitCode=0 Sep 17 18:29:43.144547 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:43.144063 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" event={"ID":"e21c0cc6-1c7b-4d20-8fa1-2f01224de923","Type":"ContainerDied","Data":"6197b2d4f63aa2cedbbfef38a88669555eddea014605a546477d7d7aa025e5fb"} Sep 17 18:29:43.144687 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:43.144668 2564 scope.go:117] "RemoveContainer" containerID="6197b2d4f63aa2cedbbfef38a88669555eddea014605a546477d7d7aa025e5fb" Sep 17 18:29:43.145887 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:43.145861 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:29:43.145959 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:43.145930 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:29:43.145997 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:43.145960 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:29:44.152860 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:44.152826 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" event={"ID":"e21c0cc6-1c7b-4d20-8fa1-2f01224de923","Type":"ContainerStarted","Data":"1c8445718c28d1f20cff28599d726d19e92176982857f2478500590e89ba3f6f"} Sep 17 18:29:44.760494 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:44.760450 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:44.760696 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:44.760661 2564 patch_prober.go:28] interesting pod/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Sep 17 18:29:44.760747 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:44.760711 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" podUID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Sep 17 18:29:45.761410 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:45.761369 2564 patch_prober.go:28] interesting pod/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Sep 17 18:29:45.761798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:45.761454 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" podUID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Sep 17 18:29:46.760469 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:46.760432 2564 patch_prober.go:28] interesting pod/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Sep 17 18:29:46.760665 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:46.760497 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" podUID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Sep 17 18:29:47.760788 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:47.760713 2564 patch_prober.go:28] interesting pod/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Sep 17 18:29:47.761154 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:47.760782 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" podUID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Sep 17 18:29:48.760655 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:48.760619 2564 patch_prober.go:28] interesting pod/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Sep 17 18:29:48.760964 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:48.760929 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" podUID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Sep 17 18:29:49.840441 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:49.840409 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:49.840944 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:49.840811 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:49.841966 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:49.841940 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:29:53.936274 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:53.936241 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8"] Sep 17 18:29:53.936676 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:53.936491 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" podUID="f521ba6e-b4b5-4372-9a32-15509e5aca8d" containerName="payload-pre-processing" containerID="cri-o://cd25901fa6e2fc33987ab39ed4bb098dbc2827d20edaff56138f8303f0fb63ac" gracePeriod=30 Sep 17 18:29:54.121705 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.121674 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7"] Sep 17 18:29:54.126444 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.126413 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b18433-xr6k7"] Sep 17 18:29:54.213335 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.213253 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" Sep 17 18:29:54.245122 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.241279 2564 generic.go:356] "Generic (PLEG): container finished" podID="f521ba6e-b4b5-4372-9a32-15509e5aca8d" containerID="cd25901fa6e2fc33987ab39ed4bb098dbc2827d20edaff56138f8303f0fb63ac" exitCode=0 Sep 17 18:29:54.245122 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.241614 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" event={"ID":"f521ba6e-b4b5-4372-9a32-15509e5aca8d","Type":"ContainerDied","Data":"cd25901fa6e2fc33987ab39ed4bb098dbc2827d20edaff56138f8303f0fb63ac"} Sep 17 18:29:54.245122 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.241679 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" Sep 17 18:29:54.245122 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.241715 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8" event={"ID":"f521ba6e-b4b5-4372-9a32-15509e5aca8d","Type":"ContainerDied","Data":"b454deef47958ad27ad51d3459c9641dd428a8cb48e2c66c6e44fec36043d442"} Sep 17 18:29:54.245122 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.241764 2564 scope.go:117] "RemoveContainer" containerID="cd25901fa6e2fc33987ab39ed4bb098dbc2827d20edaff56138f8303f0fb63ac" Sep 17 18:29:54.266364 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.266317 2564 scope.go:117] "RemoveContainer" containerID="cd25901fa6e2fc33987ab39ed4bb098dbc2827d20edaff56138f8303f0fb63ac" Sep 17 18:29:54.266961 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:29:54.266927 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd25901fa6e2fc33987ab39ed4bb098dbc2827d20edaff56138f8303f0fb63ac\": container with ID starting with cd25901fa6e2fc33987ab39ed4bb098dbc2827d20edaff56138f8303f0fb63ac not found: ID does not exist" containerID="cd25901fa6e2fc33987ab39ed4bb098dbc2827d20edaff56138f8303f0fb63ac" Sep 17 18:29:54.267052 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.266975 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd25901fa6e2fc33987ab39ed4bb098dbc2827d20edaff56138f8303f0fb63ac"} err="failed to get container status \"cd25901fa6e2fc33987ab39ed4bb098dbc2827d20edaff56138f8303f0fb63ac\": rpc error: code = NotFound desc = could not find container \"cd25901fa6e2fc33987ab39ed4bb098dbc2827d20edaff56138f8303f0fb63ac\": container with ID starting with cd25901fa6e2fc33987ab39ed4bb098dbc2827d20edaff56138f8303f0fb63ac not found: ID does not exist" Sep 17 18:29:54.299505 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.299476 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdqgn\" (UniqueName: \"kubernetes.io/projected/f521ba6e-b4b5-4372-9a32-15509e5aca8d-kube-api-access-qdqgn\") pod \"f521ba6e-b4b5-4372-9a32-15509e5aca8d\" (UID: \"f521ba6e-b4b5-4372-9a32-15509e5aca8d\") " Sep 17 18:29:54.299670 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.299556 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f521ba6e-b4b5-4372-9a32-15509e5aca8d-plugins-config-volume\") pod \"f521ba6e-b4b5-4372-9a32-15509e5aca8d\" (UID: \"f521ba6e-b4b5-4372-9a32-15509e5aca8d\") " Sep 17 18:29:54.299901 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.299880 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f521ba6e-b4b5-4372-9a32-15509e5aca8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f521ba6e-b4b5-4372-9a32-15509e5aca8d" (UID: "f521ba6e-b4b5-4372-9a32-15509e5aca8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:29:54.299986 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.299967 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f521ba6e-b4b5-4372-9a32-15509e5aca8d-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:29:54.301792 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.301771 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f521ba6e-b4b5-4372-9a32-15509e5aca8d-kube-api-access-qdqgn" (OuterVolumeSpecName: "kube-api-access-qdqgn") pod "f521ba6e-b4b5-4372-9a32-15509e5aca8d" (UID: "f521ba6e-b4b5-4372-9a32-15509e5aca8d"). InnerVolumeSpecName "kube-api-access-qdqgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:29:54.401064 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.401036 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdqgn\" (UniqueName: \"kubernetes.io/projected/f521ba6e-b4b5-4372-9a32-15509e5aca8d-kube-api-access-qdqgn\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:29:54.565539 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.565502 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8"] Sep 17 18:29:54.573105 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:54.573083 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b18433-5d46fd8dbb-p8ws8"] Sep 17 18:29:55.451884 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:55.451855 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="607a95a0-8f20-474a-b841-385ca1d957c2" path="/var/lib/kubelet/pods/607a95a0-8f20-474a-b841-385ca1d957c2/volumes" Sep 17 18:29:55.452236 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:29:55.452215 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f521ba6e-b4b5-4372-9a32-15509e5aca8d" path="/var/lib/kubelet/pods/f521ba6e-b4b5-4372-9a32-15509e5aca8d/volumes" Sep 17 18:30:00.153599 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.153567 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2"] Sep 17 18:30:00.154096 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.154080 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="607a95a0-8f20-474a-b841-385ca1d957c2" containerName="revoke-keys" Sep 17 18:30:00.154141 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.154098 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="607a95a0-8f20-474a-b841-385ca1d957c2" containerName="revoke-keys" Sep 17 18:30:00.154141 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.154126 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f521ba6e-b4b5-4372-9a32-15509e5aca8d" containerName="payload-pre-processing" Sep 17 18:30:00.154141 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.154132 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f521ba6e-b4b5-4372-9a32-15509e5aca8d" containerName="payload-pre-processing" Sep 17 18:30:00.154243 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.154203 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f521ba6e-b4b5-4372-9a32-15509e5aca8d" containerName="payload-pre-processing" Sep 17 18:30:00.154243 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.154214 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="607a95a0-8f20-474a-b841-385ca1d957c2" containerName="revoke-keys" Sep 17 18:30:00.156401 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.156379 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj"] Sep 17 18:30:00.156459 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.156402 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2" Sep 17 18:30:00.159247 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.159228 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj" Sep 17 18:30:00.159375 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.159315 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\"" Sep 17 18:30:00.167600 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.167569 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2" Sep 17 18:30:00.172655 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.172629 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2"] Sep 17 18:30:00.174878 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.174854 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj" Sep 17 18:30:00.180578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.180556 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj"] Sep 17 18:30:00.186976 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.186947 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd"] Sep 17 18:30:00.191240 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.191190 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd" Sep 17 18:30:00.213303 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.213232 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd"] Sep 17 18:30:00.235302 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.235263 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd" Sep 17 18:30:00.471286 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.471259 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2"] Sep 17 18:30:00.471836 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:30:00.471814 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12795c6d_6d9c_4ce8_a6be_decbb9f9af6a.slice/crio-123052904efbfa6630e7c16e8ff115a1de8da97339778b6be92c5f558877be79 WatchSource:0}: Error finding container 123052904efbfa6630e7c16e8ff115a1de8da97339778b6be92c5f558877be79: Status 404 returned error can't find the container with id 123052904efbfa6630e7c16e8ff115a1de8da97339778b6be92c5f558877be79 Sep 17 18:30:00.489577 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:30:00.489549 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb50ff34_9c97_4f32_870a_a120e2bd7681.slice/crio-d575c1f0a64e2f08c2359d63b9abf2eedad2b62d7daf583e2f73a54581d834bb WatchSource:0}: Error finding container d575c1f0a64e2f08c2359d63b9abf2eedad2b62d7daf583e2f73a54581d834bb: Status 404 returned error can't find the container with id d575c1f0a64e2f08c2359d63b9abf2eedad2b62d7daf583e2f73a54581d834bb Sep 17 18:30:00.489823 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.489797 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj"] Sep 17 18:30:00.551824 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:00.551801 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd"] Sep 17 18:30:00.553907 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:30:00.553887 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b55eacb_6168_4165_8001_c2827eeb1efa.slice/crio-107abcc2518d9449b77ac52623c5140b55f52f5f33acf29b2d9f883404d18c34 WatchSource:0}: Error finding container 107abcc2518d9449b77ac52623c5140b55f52f5f33acf29b2d9f883404d18c34: Status 404 returned error can't find the container with id 107abcc2518d9449b77ac52623c5140b55f52f5f33acf29b2d9f883404d18c34 Sep 17 18:30:01.293588 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:01.293500 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd" event={"ID":"8b55eacb-6168-4165-8001-c2827eeb1efa","Type":"ContainerStarted","Data":"75009309fac3badf1fc686fc542e0ab73680ed82b97f410a0eff14cbadfe669b"} Sep 17 18:30:01.294101 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:01.293612 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd" event={"ID":"8b55eacb-6168-4165-8001-c2827eeb1efa","Type":"ContainerStarted","Data":"107abcc2518d9449b77ac52623c5140b55f52f5f33acf29b2d9f883404d18c34"} Sep 17 18:30:01.295194 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:01.295168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj" event={"ID":"eb50ff34-9c97-4f32-870a-a120e2bd7681","Type":"ContainerStarted","Data":"adb62eb4763c3411e02b97e651bd5d09e3ff499ec2b206783fa67faf46beed6a"} Sep 17 18:30:01.295324 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:01.295210 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj" event={"ID":"eb50ff34-9c97-4f32-870a-a120e2bd7681","Type":"ContainerStarted","Data":"d575c1f0a64e2f08c2359d63b9abf2eedad2b62d7daf583e2f73a54581d834bb"} Sep 17 18:30:01.297220 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:01.297180 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2" event={"ID":"12795c6d-6d9c-4ce8-a6be-decbb9f9af6a","Type":"ContainerStarted","Data":"c79fc153b65aabdcc35c0651b8641acb9008d7fb7387fbbc2dfc7c1b14c9a96d"} Sep 17 18:30:01.297327 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:01.297241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2" event={"ID":"12795c6d-6d9c-4ce8-a6be-decbb9f9af6a","Type":"ContainerStarted","Data":"123052904efbfa6630e7c16e8ff115a1de8da97339778b6be92c5f558877be79"} Sep 17 18:30:01.312407 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:01.312364 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd" podStartSLOduration=1.312349975 podStartE2EDuration="1.312349975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.311108146 +0000 UTC m=+1626.321295283" watchObservedRunningTime="2026-09-17 18:30:01.312349975 +0000 UTC m=+1626.322537141" Sep 17 18:30:01.330307 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:01.330265 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2" podStartSLOduration=1.33024647 podStartE2EDuration="1.33024647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.327433755 +0000 UTC m=+1626.337620892" watchObservedRunningTime="2026-09-17 18:30:01.33024647 +0000 UTC m=+1626.340433606" Sep 17 18:30:01.346331 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:01.346289 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj" podStartSLOduration=1.346273193 podStartE2EDuration="1.346273193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.344984366 +0000 UTC m=+1626.355171502" watchObservedRunningTime="2026-09-17 18:30:01.346273193 +0000 UTC m=+1626.356460330" Sep 17 18:30:02.307399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:02.307366 2564 generic.go:356] "Generic (PLEG): container finished" podID="8b55eacb-6168-4165-8001-c2827eeb1efa" containerID="75009309fac3badf1fc686fc542e0ab73680ed82b97f410a0eff14cbadfe669b" exitCode=0 Sep 17 18:30:02.307808 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:02.307417 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd" event={"ID":"8b55eacb-6168-4165-8001-c2827eeb1efa","Type":"ContainerDied","Data":"75009309fac3badf1fc686fc542e0ab73680ed82b97f410a0eff14cbadfe669b"} Sep 17 18:30:02.309235 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:02.309206 2564 generic.go:356] "Generic (PLEG): container finished" podID="eb50ff34-9c97-4f32-870a-a120e2bd7681" containerID="adb62eb4763c3411e02b97e651bd5d09e3ff499ec2b206783fa67faf46beed6a" exitCode=0 Sep 17 18:30:02.309368 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:02.309285 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj" event={"ID":"eb50ff34-9c97-4f32-870a-a120e2bd7681","Type":"ContainerDied","Data":"adb62eb4763c3411e02b97e651bd5d09e3ff499ec2b206783fa67faf46beed6a"} Sep 17 18:30:02.310931 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:02.310911 2564 generic.go:356] "Generic (PLEG): container finished" podID="12795c6d-6d9c-4ce8-a6be-decbb9f9af6a" containerID="c79fc153b65aabdcc35c0651b8641acb9008d7fb7387fbbc2dfc7c1b14c9a96d" exitCode=0 Sep 17 18:30:02.311021 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:02.310944 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2" event={"ID":"12795c6d-6d9c-4ce8-a6be-decbb9f9af6a","Type":"ContainerDied","Data":"c79fc153b65aabdcc35c0651b8641acb9008d7fb7387fbbc2dfc7c1b14c9a96d"} Sep 17 18:30:02.459285 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:02.459252 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9"] Sep 17 18:30:02.459487 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:02.459453 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" containerID="cri-o://401987bba49844afae4700deab854b3860c144acadc1e103827065724b71a2fe" gracePeriod=30 Sep 17 18:30:03.515369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:03.515344 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd" Sep 17 18:30:03.550246 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:03.550218 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2" Sep 17 18:30:03.553777 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:03.553755 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj" Sep 17 18:30:04.324841 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:04.324813 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj" Sep 17 18:30:04.325004 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:04.324812 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj" event={"ID":"eb50ff34-9c97-4f32-870a-a120e2bd7681","Type":"ContainerDied","Data":"d575c1f0a64e2f08c2359d63b9abf2eedad2b62d7daf583e2f73a54581d834bb"} Sep 17 18:30:04.325004 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:04.324941 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d575c1f0a64e2f08c2359d63b9abf2eedad2b62d7daf583e2f73a54581d834bb" Sep 17 18:30:04.326341 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:04.326320 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2" Sep 17 18:30:04.326471 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:04.326363 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2" event={"ID":"12795c6d-6d9c-4ce8-a6be-decbb9f9af6a","Type":"ContainerDied","Data":"123052904efbfa6630e7c16e8ff115a1de8da97339778b6be92c5f558877be79"} Sep 17 18:30:04.326471 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:04.326404 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="123052904efbfa6630e7c16e8ff115a1de8da97339778b6be92c5f558877be79" Sep 17 18:30:04.327975 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:04.327955 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd" event={"ID":"8b55eacb-6168-4165-8001-c2827eeb1efa","Type":"ContainerDied","Data":"107abcc2518d9449b77ac52623c5140b55f52f5f33acf29b2d9f883404d18c34"} Sep 17 18:30:04.328069 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:04.327994 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="107abcc2518d9449b77ac52623c5140b55f52f5f33acf29b2d9f883404d18c34" Sep 17 18:30:04.328069 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:04.327956 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd" Sep 17 18:30:07.753895 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.753868 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:30:07.845346 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.845318 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-podinfo\") pod \"71ef44e5-6376-4ab4-960f-d6fa1104e301\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " Sep 17 18:30:07.845346 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.845351 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-workload-socket\") pod \"71ef44e5-6376-4ab4-960f-d6fa1104e301\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " Sep 17 18:30:07.845599 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.845414 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45n7m\" (UniqueName: \"kubernetes.io/projected/71ef44e5-6376-4ab4-960f-d6fa1104e301-kube-api-access-45n7m\") pod \"71ef44e5-6376-4ab4-960f-d6fa1104e301\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " Sep 17 18:30:07.845599 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.845445 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-envoy\") pod \"71ef44e5-6376-4ab4-960f-d6fa1104e301\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " Sep 17 18:30:07.845599 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.845462 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-token\") pod \"71ef44e5-6376-4ab4-960f-d6fa1104e301\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " Sep 17 18:30:07.845599 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.845484 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ef44e5-6376-4ab4-960f-d6fa1104e301-istiod-ca-cert\") pod \"71ef44e5-6376-4ab4-960f-d6fa1104e301\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " Sep 17 18:30:07.845599 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.845507 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-data\") pod \"71ef44e5-6376-4ab4-960f-d6fa1104e301\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " Sep 17 18:30:07.845599 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.845586 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-credential-socket\") pod \"71ef44e5-6376-4ab4-960f-d6fa1104e301\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " Sep 17 18:30:07.845920 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.845657 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-workload-certs\") pod \"71ef44e5-6376-4ab4-960f-d6fa1104e301\" (UID: \"71ef44e5-6376-4ab4-960f-d6fa1104e301\") " Sep 17 18:30:07.845920 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.845826 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71ef44e5-6376-4ab4-960f-d6fa1104e301" (UID: "71ef44e5-6376-4ab4-960f-d6fa1104e301"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:07.845920 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.845885 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-data" (OuterVolumeSpecName: "istio-data") pod "71ef44e5-6376-4ab4-960f-d6fa1104e301" (UID: "71ef44e5-6376-4ab4-960f-d6fa1104e301"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:07.846074 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.845949 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ef44e5-6376-4ab4-960f-d6fa1104e301-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71ef44e5-6376-4ab4-960f-d6fa1104e301" (UID: "71ef44e5-6376-4ab4-960f-d6fa1104e301"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:07.846074 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.846039 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71ef44e5-6376-4ab4-960f-d6fa1104e301" (UID: "71ef44e5-6376-4ab4-960f-d6fa1104e301"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:07.846347 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.846082 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71ef44e5-6376-4ab4-960f-d6fa1104e301" (UID: "71ef44e5-6376-4ab4-960f-d6fa1104e301"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:07.846347 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.846148 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ef44e5-6376-4ab4-960f-d6fa1104e301-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:07.846347 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.846161 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:07.846347 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.846169 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:07.846347 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.846177 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:07.847859 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.847831 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71ef44e5-6376-4ab4-960f-d6fa1104e301" (UID: "71ef44e5-6376-4ab4-960f-d6fa1104e301"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:07.847974 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.847890 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71ef44e5-6376-4ab4-960f-d6fa1104e301" (UID: "71ef44e5-6376-4ab4-960f-d6fa1104e301"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:30:07.847974 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.847909 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-token" (OuterVolumeSpecName: "istio-token") pod "71ef44e5-6376-4ab4-960f-d6fa1104e301" (UID: "71ef44e5-6376-4ab4-960f-d6fa1104e301"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:07.848148 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.848122 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ef44e5-6376-4ab4-960f-d6fa1104e301-kube-api-access-45n7m" (OuterVolumeSpecName: "kube-api-access-45n7m") pod "71ef44e5-6376-4ab4-960f-d6fa1104e301" (UID: "71ef44e5-6376-4ab4-960f-d6fa1104e301"). InnerVolumeSpecName "kube-api-access-45n7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:07.947031 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.947005 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45n7m\" (UniqueName: \"kubernetes.io/projected/71ef44e5-6376-4ab4-960f-d6fa1104e301-kube-api-access-45n7m\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:07.947031 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.947027 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:07.947176 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.947038 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:07.947176 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.947046 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef44e5-6376-4ab4-960f-d6fa1104e301-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:07.947176 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:07.947054 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ef44e5-6376-4ab4-960f-d6fa1104e301-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:08.355752 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:08.355717 2564 generic.go:356] "Generic (PLEG): container finished" podID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerID="401987bba49844afae4700deab854b3860c144acadc1e103827065724b71a2fe" exitCode=0 Sep 17 18:30:08.355927 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:08.355764 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" Sep 17 18:30:08.355927 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:08.355801 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" event={"ID":"71ef44e5-6376-4ab4-960f-d6fa1104e301","Type":"ContainerDied","Data":"401987bba49844afae4700deab854b3860c144acadc1e103827065724b71a2fe"} Sep 17 18:30:08.355927 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:08.355861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9" event={"ID":"71ef44e5-6376-4ab4-960f-d6fa1104e301","Type":"ContainerDied","Data":"774f402280d0a791207c9fe6fd8953e688f48c5713be566d918ef197f9c091cb"} Sep 17 18:30:08.355927 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:08.355891 2564 scope.go:117] "RemoveContainer" containerID="401987bba49844afae4700deab854b3860c144acadc1e103827065724b71a2fe" Sep 17 18:30:08.366726 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:08.366706 2564 scope.go:117] "RemoveContainer" containerID="5785402cb65d6afde4676d10790e52c7e2cb86c915ec615fdd1182e8530565ad" Sep 17 18:30:08.375221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:08.375204 2564 scope.go:117] "RemoveContainer" containerID="401987bba49844afae4700deab854b3860c144acadc1e103827065724b71a2fe" Sep 17 18:30:08.375500 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:08.375481 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401987bba49844afae4700deab854b3860c144acadc1e103827065724b71a2fe\": container with ID starting with 401987bba49844afae4700deab854b3860c144acadc1e103827065724b71a2fe not found: ID does not exist" containerID="401987bba49844afae4700deab854b3860c144acadc1e103827065724b71a2fe" Sep 17 18:30:08.375578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:08.375507 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401987bba49844afae4700deab854b3860c144acadc1e103827065724b71a2fe"} err="failed to get container status \"401987bba49844afae4700deab854b3860c144acadc1e103827065724b71a2fe\": rpc error: code = NotFound desc = could not find container \"401987bba49844afae4700deab854b3860c144acadc1e103827065724b71a2fe\": container with ID starting with 401987bba49844afae4700deab854b3860c144acadc1e103827065724b71a2fe not found: ID does not exist" Sep 17 18:30:08.375578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:08.375539 2564 scope.go:117] "RemoveContainer" containerID="5785402cb65d6afde4676d10790e52c7e2cb86c915ec615fdd1182e8530565ad" Sep 17 18:30:08.376081 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:08.376015 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5785402cb65d6afde4676d10790e52c7e2cb86c915ec615fdd1182e8530565ad\": container with ID starting with 5785402cb65d6afde4676d10790e52c7e2cb86c915ec615fdd1182e8530565ad not found: ID does not exist" containerID="5785402cb65d6afde4676d10790e52c7e2cb86c915ec615fdd1182e8530565ad" Sep 17 18:30:08.376081 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:08.376051 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5785402cb65d6afde4676d10790e52c7e2cb86c915ec615fdd1182e8530565ad"} err="failed to get container status \"5785402cb65d6afde4676d10790e52c7e2cb86c915ec615fdd1182e8530565ad\": rpc error: code = NotFound desc = could not find container \"5785402cb65d6afde4676d10790e52c7e2cb86c915ec615fdd1182e8530565ad\": container with ID starting with 5785402cb65d6afde4676d10790e52c7e2cb86c915ec615fdd1182e8530565ad not found: ID does not exist" Sep 17 18:30:08.379318 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:08.379296 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9"] Sep 17 18:30:08.383981 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:08.383960 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b18433-openshift-default-7588884dfd-pfcv9"] Sep 17 18:30:09.451493 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:09.451451 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" path="/var/lib/kubelet/pods/71ef44e5-6376-4ab4-960f-d6fa1104e301/volumes" Sep 17 18:30:18.315429 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:18.315399 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb"] Sep 17 18:30:20.306042 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:20.306008 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd"] Sep 17 18:30:20.308714 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:20.308689 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b06908-29827830-xg5jd"] Sep 17 18:30:20.346253 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:20.346225 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx"] Sep 17 18:30:20.346519 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:20.346462 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" podUID="86d6d36a-bc35-4381-a8bd-9badc7e3d03a" containerName="payload-pre-processing" containerID="cri-o://f73a853b7f5495602727bc132a8aa69a830d51636f8ee1b47c70e7cdeb450fe9" gracePeriod=30 Sep 17 18:30:20.605273 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:20.605247 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" Sep 17 18:30:20.767461 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:20.767418 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2msv\" (UniqueName: \"kubernetes.io/projected/86d6d36a-bc35-4381-a8bd-9badc7e3d03a-kube-api-access-m2msv\") pod \"86d6d36a-bc35-4381-a8bd-9badc7e3d03a\" (UID: \"86d6d36a-bc35-4381-a8bd-9badc7e3d03a\") " Sep 17 18:30:20.767701 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:20.767489 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d6d36a-bc35-4381-a8bd-9badc7e3d03a-plugins-config-volume\") pod \"86d6d36a-bc35-4381-a8bd-9badc7e3d03a\" (UID: \"86d6d36a-bc35-4381-a8bd-9badc7e3d03a\") " Sep 17 18:30:20.768305 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:20.768269 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d6d36a-bc35-4381-a8bd-9badc7e3d03a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86d6d36a-bc35-4381-a8bd-9badc7e3d03a" (UID: "86d6d36a-bc35-4381-a8bd-9badc7e3d03a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:20.771322 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:20.771293 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d6d36a-bc35-4381-a8bd-9badc7e3d03a-kube-api-access-m2msv" (OuterVolumeSpecName: "kube-api-access-m2msv") pod "86d6d36a-bc35-4381-a8bd-9badc7e3d03a" (UID: "86d6d36a-bc35-4381-a8bd-9badc7e3d03a"). InnerVolumeSpecName "kube-api-access-m2msv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:20.869492 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:20.869405 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2msv\" (UniqueName: \"kubernetes.io/projected/86d6d36a-bc35-4381-a8bd-9badc7e3d03a-kube-api-access-m2msv\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:20.869492 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:20.869437 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d6d36a-bc35-4381-a8bd-9badc7e3d03a-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:21.442258 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:21.442225 2564 generic.go:356] "Generic (PLEG): container finished" podID="86d6d36a-bc35-4381-a8bd-9badc7e3d03a" containerID="f73a853b7f5495602727bc132a8aa69a830d51636f8ee1b47c70e7cdeb450fe9" exitCode=0 Sep 17 18:30:21.442683 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:21.442270 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" Sep 17 18:30:21.442683 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:21.442281 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" event={"ID":"86d6d36a-bc35-4381-a8bd-9badc7e3d03a","Type":"ContainerDied","Data":"f73a853b7f5495602727bc132a8aa69a830d51636f8ee1b47c70e7cdeb450fe9"} Sep 17 18:30:21.442683 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:21.442320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx" event={"ID":"86d6d36a-bc35-4381-a8bd-9badc7e3d03a","Type":"ContainerDied","Data":"7f5f52d416e2bd9bbe3716a29a2c3ddd538ec5afbbc07615a393dbc2bebb73b5"} Sep 17 18:30:21.442683 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:21.442342 2564 scope.go:117] "RemoveContainer" containerID="f73a853b7f5495602727bc132a8aa69a830d51636f8ee1b47c70e7cdeb450fe9" Sep 17 18:30:21.452412 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:21.452385 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b55eacb-6168-4165-8001-c2827eeb1efa" path="/var/lib/kubelet/pods/8b55eacb-6168-4165-8001-c2827eeb1efa/volumes" Sep 17 18:30:21.453607 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:21.453586 2564 scope.go:117] "RemoveContainer" containerID="f73a853b7f5495602727bc132a8aa69a830d51636f8ee1b47c70e7cdeb450fe9" Sep 17 18:30:21.453902 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:21.453872 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73a853b7f5495602727bc132a8aa69a830d51636f8ee1b47c70e7cdeb450fe9\": container with ID starting with f73a853b7f5495602727bc132a8aa69a830d51636f8ee1b47c70e7cdeb450fe9 not found: ID does not exist" containerID="f73a853b7f5495602727bc132a8aa69a830d51636f8ee1b47c70e7cdeb450fe9" Sep 17 18:30:21.453979 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:21.453914 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73a853b7f5495602727bc132a8aa69a830d51636f8ee1b47c70e7cdeb450fe9"} err="failed to get container status \"f73a853b7f5495602727bc132a8aa69a830d51636f8ee1b47c70e7cdeb450fe9\": rpc error: code = NotFound desc = could not find container \"f73a853b7f5495602727bc132a8aa69a830d51636f8ee1b47c70e7cdeb450fe9\": container with ID starting with f73a853b7f5495602727bc132a8aa69a830d51636f8ee1b47c70e7cdeb450fe9 not found: ID does not exist" Sep 17 18:30:21.465251 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:21.465224 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx"] Sep 17 18:30:21.472508 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:21.472486 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b06908-7dd968cb9d-8cglx"] Sep 17 18:30:23.452407 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:23.452374 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86d6d36a-bc35-4381-a8bd-9badc7e3d03a" path="/var/lib/kubelet/pods/86d6d36a-bc35-4381-a8bd-9badc7e3d03a/volumes" Sep 17 18:30:27.428815 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:27.428785 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb"] Sep 17 18:30:27.429286 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:27.429024 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" podUID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerName="istio-proxy" containerID="cri-o://1c8445718c28d1f20cff28599d726d19e92176982857f2478500590e89ba3f6f" gracePeriod=30 Sep 17 18:30:32.055799 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.055755 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48"] Sep 17 18:30:32.056280 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.055993 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" podUID="0bde362a-db00-456e-954d-263597ad8c1e" containerName="payload-processing" containerID="cri-o://09f223bfe43074f18513c53dd9fd47629ce6b0c02e3d79e91874589396d68964" gracePeriod=30 Sep 17 18:30:32.062998 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.062968 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g"] Sep 17 18:30:32.063243 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.063195 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" podUID="48c73ee8-66dd-4f6e-9cf9-87696e8cc69b" containerName="payload-pre-processing" containerID="cri-o://9fe98b2d7c69ac1bbdc828b02fd8a2d89485a8ad0b1cb31e71ab9baf1587916c" gracePeriod=30 Sep 17 18:30:32.380718 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.380693 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" Sep 17 18:30:32.383722 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.383704 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" Sep 17 18:30:32.474222 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.474201 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bde362a-db00-456e-954d-263597ad8c1e-plugins-config-volume\") pod \"0bde362a-db00-456e-954d-263597ad8c1e\" (UID: \"0bde362a-db00-456e-954d-263597ad8c1e\") " Sep 17 18:30:32.474328 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.474247 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6b9b\" (UniqueName: \"kubernetes.io/projected/48c73ee8-66dd-4f6e-9cf9-87696e8cc69b-kube-api-access-k6b9b\") pod \"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b\" (UID: \"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b\") " Sep 17 18:30:32.474328 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.474305 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v9cm\" (UniqueName: \"kubernetes.io/projected/0bde362a-db00-456e-954d-263597ad8c1e-kube-api-access-7v9cm\") pod \"0bde362a-db00-456e-954d-263597ad8c1e\" (UID: \"0bde362a-db00-456e-954d-263597ad8c1e\") " Sep 17 18:30:32.474408 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.474371 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c73ee8-66dd-4f6e-9cf9-87696e8cc69b-plugins-config-volume\") pod \"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b\" (UID: \"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b\") " Sep 17 18:30:32.474615 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.474591 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bde362a-db00-456e-954d-263597ad8c1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0bde362a-db00-456e-954d-263597ad8c1e" (UID: "0bde362a-db00-456e-954d-263597ad8c1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:32.474747 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.474725 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c73ee8-66dd-4f6e-9cf9-87696e8cc69b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "48c73ee8-66dd-4f6e-9cf9-87696e8cc69b" (UID: "48c73ee8-66dd-4f6e-9cf9-87696e8cc69b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:32.474925 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.474910 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c73ee8-66dd-4f6e-9cf9-87696e8cc69b-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.474993 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.474931 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bde362a-db00-456e-954d-263597ad8c1e-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.476581 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.476559 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c73ee8-66dd-4f6e-9cf9-87696e8cc69b-kube-api-access-k6b9b" (OuterVolumeSpecName: "kube-api-access-k6b9b") pod "48c73ee8-66dd-4f6e-9cf9-87696e8cc69b" (UID: "48c73ee8-66dd-4f6e-9cf9-87696e8cc69b"). InnerVolumeSpecName "kube-api-access-k6b9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:32.476648 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.476620 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bde362a-db00-456e-954d-263597ad8c1e-kube-api-access-7v9cm" (OuterVolumeSpecName: "kube-api-access-7v9cm") pod "0bde362a-db00-456e-954d-263597ad8c1e" (UID: "0bde362a-db00-456e-954d-263597ad8c1e"). InnerVolumeSpecName "kube-api-access-7v9cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:32.529186 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.529156 2564 generic.go:356] "Generic (PLEG): container finished" podID="48c73ee8-66dd-4f6e-9cf9-87696e8cc69b" containerID="9fe98b2d7c69ac1bbdc828b02fd8a2d89485a8ad0b1cb31e71ab9baf1587916c" exitCode=0 Sep 17 18:30:32.529289 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.529204 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" Sep 17 18:30:32.529289 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.529210 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" event={"ID":"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b","Type":"ContainerDied","Data":"9fe98b2d7c69ac1bbdc828b02fd8a2d89485a8ad0b1cb31e71ab9baf1587916c"} Sep 17 18:30:32.529289 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.529269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g" event={"ID":"48c73ee8-66dd-4f6e-9cf9-87696e8cc69b","Type":"ContainerDied","Data":"15709a562c0dd829deb20e358b9af8623b511829f04ec009f0294bc96e090beb"} Sep 17 18:30:32.529432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.529296 2564 scope.go:117] "RemoveContainer" containerID="9fe98b2d7c69ac1bbdc828b02fd8a2d89485a8ad0b1cb31e71ab9baf1587916c" Sep 17 18:30:32.530820 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.530794 2564 generic.go:356] "Generic (PLEG): container finished" podID="0bde362a-db00-456e-954d-263597ad8c1e" containerID="09f223bfe43074f18513c53dd9fd47629ce6b0c02e3d79e91874589396d68964" exitCode=0 Sep 17 18:30:32.530909 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.530890 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" Sep 17 18:30:32.530958 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.530936 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" event={"ID":"0bde362a-db00-456e-954d-263597ad8c1e","Type":"ContainerDied","Data":"09f223bfe43074f18513c53dd9fd47629ce6b0c02e3d79e91874589396d68964"} Sep 17 18:30:32.530996 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.530977 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48" event={"ID":"0bde362a-db00-456e-954d-263597ad8c1e","Type":"ContainerDied","Data":"6cf9bb8b21fb622e44d1ec338573f3a17dbd9e46ee034c18385672aefd52dd24"} Sep 17 18:30:32.542895 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.542874 2564 scope.go:117] "RemoveContainer" containerID="9fe98b2d7c69ac1bbdc828b02fd8a2d89485a8ad0b1cb31e71ab9baf1587916c" Sep 17 18:30:32.543166 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:32.543144 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fe98b2d7c69ac1bbdc828b02fd8a2d89485a8ad0b1cb31e71ab9baf1587916c\": container with ID starting with 9fe98b2d7c69ac1bbdc828b02fd8a2d89485a8ad0b1cb31e71ab9baf1587916c not found: ID does not exist" containerID="9fe98b2d7c69ac1bbdc828b02fd8a2d89485a8ad0b1cb31e71ab9baf1587916c" Sep 17 18:30:32.543229 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.543174 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fe98b2d7c69ac1bbdc828b02fd8a2d89485a8ad0b1cb31e71ab9baf1587916c"} err="failed to get container status \"9fe98b2d7c69ac1bbdc828b02fd8a2d89485a8ad0b1cb31e71ab9baf1587916c\": rpc error: code = NotFound desc = could not find container \"9fe98b2d7c69ac1bbdc828b02fd8a2d89485a8ad0b1cb31e71ab9baf1587916c\": container with ID starting with 9fe98b2d7c69ac1bbdc828b02fd8a2d89485a8ad0b1cb31e71ab9baf1587916c not found: ID does not exist" Sep 17 18:30:32.543229 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.543193 2564 scope.go:117] "RemoveContainer" containerID="09f223bfe43074f18513c53dd9fd47629ce6b0c02e3d79e91874589396d68964" Sep 17 18:30:32.561428 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.560939 2564 scope.go:117] "RemoveContainer" containerID="09f223bfe43074f18513c53dd9fd47629ce6b0c02e3d79e91874589396d68964" Sep 17 18:30:32.567344 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:32.563284 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f223bfe43074f18513c53dd9fd47629ce6b0c02e3d79e91874589396d68964\": container with ID starting with 09f223bfe43074f18513c53dd9fd47629ce6b0c02e3d79e91874589396d68964 not found: ID does not exist" containerID="09f223bfe43074f18513c53dd9fd47629ce6b0c02e3d79e91874589396d68964" Sep 17 18:30:32.567344 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.563337 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f223bfe43074f18513c53dd9fd47629ce6b0c02e3d79e91874589396d68964"} err="failed to get container status \"09f223bfe43074f18513c53dd9fd47629ce6b0c02e3d79e91874589396d68964\": rpc error: code = NotFound desc = could not find container \"09f223bfe43074f18513c53dd9fd47629ce6b0c02e3d79e91874589396d68964\": container with ID starting with 09f223bfe43074f18513c53dd9fd47629ce6b0c02e3d79e91874589396d68964 not found: ID does not exist" Sep 17 18:30:32.567798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.567761 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48"] Sep 17 18:30:32.573821 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.573740 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5d8053-6b4b7f7cbf-tmw48"] Sep 17 18:30:32.576498 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.576477 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6b9b\" (UniqueName: \"kubernetes.io/projected/48c73ee8-66dd-4f6e-9cf9-87696e8cc69b-kube-api-access-k6b9b\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.576498 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.576497 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v9cm\" (UniqueName: \"kubernetes.io/projected/0bde362a-db00-456e-954d-263597ad8c1e-kube-api-access-7v9cm\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.593877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.593846 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g"] Sep 17 18:30:32.609657 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.609633 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5d8053-74f9f9cf86-k464g"] Sep 17 18:30:32.688329 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.688309 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:30:32.879260 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.879184 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-data\") pod \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " Sep 17 18:30:32.879260 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.879218 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5c9t\" (UniqueName: \"kubernetes.io/projected/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-kube-api-access-z5c9t\") pod \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " Sep 17 18:30:32.879486 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.879261 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istiod-ca-cert\") pod \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " Sep 17 18:30:32.879486 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.879290 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-envoy\") pod \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " Sep 17 18:30:32.879486 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.879341 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-workload-certs\") pod \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " Sep 17 18:30:32.879486 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.879392 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-token\") pod \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " Sep 17 18:30:32.879486 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.879464 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-credential-socket\") pod \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " Sep 17 18:30:32.879790 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.879483 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-data" (OuterVolumeSpecName: "istio-data") pod "e21c0cc6-1c7b-4d20-8fa1-2f01224de923" (UID: "e21c0cc6-1c7b-4d20-8fa1-2f01224de923"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:32.879790 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.879571 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-workload-socket\") pod \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " Sep 17 18:30:32.879790 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.879640 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-podinfo\") pod \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\" (UID: \"e21c0cc6-1c7b-4d20-8fa1-2f01224de923\") " Sep 17 18:30:32.879790 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.879686 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e21c0cc6-1c7b-4d20-8fa1-2f01224de923" (UID: "e21c0cc6-1c7b-4d20-8fa1-2f01224de923"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:32.879790 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.879759 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e21c0cc6-1c7b-4d20-8fa1-2f01224de923" (UID: "e21c0cc6-1c7b-4d20-8fa1-2f01224de923"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:32.880018 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.879870 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e21c0cc6-1c7b-4d20-8fa1-2f01224de923" (UID: "e21c0cc6-1c7b-4d20-8fa1-2f01224de923"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:32.880314 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.880285 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e21c0cc6-1c7b-4d20-8fa1-2f01224de923" (UID: "e21c0cc6-1c7b-4d20-8fa1-2f01224de923"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:32.880406 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.880370 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.880406 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.880393 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.880482 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.880409 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.880482 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.880422 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.881591 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.881570 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-kube-api-access-z5c9t" (OuterVolumeSpecName: "kube-api-access-z5c9t") pod "e21c0cc6-1c7b-4d20-8fa1-2f01224de923" (UID: "e21c0cc6-1c7b-4d20-8fa1-2f01224de923"). InnerVolumeSpecName "kube-api-access-z5c9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:32.881746 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.881731 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e21c0cc6-1c7b-4d20-8fa1-2f01224de923" (UID: "e21c0cc6-1c7b-4d20-8fa1-2f01224de923"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:30:32.882164 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.882144 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e21c0cc6-1c7b-4d20-8fa1-2f01224de923" (UID: "e21c0cc6-1c7b-4d20-8fa1-2f01224de923"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:32.882234 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.882166 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-token" (OuterVolumeSpecName: "istio-token") pod "e21c0cc6-1c7b-4d20-8fa1-2f01224de923" (UID: "e21c0cc6-1c7b-4d20-8fa1-2f01224de923"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:32.981614 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.981591 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5c9t\" (UniqueName: \"kubernetes.io/projected/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-kube-api-access-z5c9t\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.981614 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.981614 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.981757 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.981623 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.981757 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.981631 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.981757 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:32.981640 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21c0cc6-1c7b-4d20-8fa1-2f01224de923-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:33.335916 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.335888 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" podUID="b14b32c9-c9df-4054-a0bd-6f9a9dc676e2" containerName="main" containerID="cri-o://296c94ea45793c73071c86bbeb2e7efef09a8a3b55ec8f7c514aa1aafe4788b2" gracePeriod=45 Sep 17 18:30:33.451633 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.451607 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bde362a-db00-456e-954d-263597ad8c1e" path="/var/lib/kubelet/pods/0bde362a-db00-456e-954d-263597ad8c1e/volumes" Sep 17 18:30:33.452011 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.451995 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48c73ee8-66dd-4f6e-9cf9-87696e8cc69b" path="/var/lib/kubelet/pods/48c73ee8-66dd-4f6e-9cf9-87696e8cc69b/volumes" Sep 17 18:30:33.543095 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.543066 2564 generic.go:356] "Generic (PLEG): container finished" podID="b14b32c9-c9df-4054-a0bd-6f9a9dc676e2" containerID="296c94ea45793c73071c86bbeb2e7efef09a8a3b55ec8f7c514aa1aafe4788b2" exitCode=0 Sep 17 18:30:33.543226 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.543143 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" event={"ID":"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2","Type":"ContainerDied","Data":"296c94ea45793c73071c86bbeb2e7efef09a8a3b55ec8f7c514aa1aafe4788b2"} Sep 17 18:30:33.546255 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.546219 2564 generic.go:356] "Generic (PLEG): container finished" podID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerID="1c8445718c28d1f20cff28599d726d19e92176982857f2478500590e89ba3f6f" exitCode=0 Sep 17 18:30:33.546361 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.546293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" event={"ID":"e21c0cc6-1c7b-4d20-8fa1-2f01224de923","Type":"ContainerDied","Data":"1c8445718c28d1f20cff28599d726d19e92176982857f2478500590e89ba3f6f"} Sep 17 18:30:33.546361 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.546311 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" Sep 17 18:30:33.546361 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.546343 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb" event={"ID":"e21c0cc6-1c7b-4d20-8fa1-2f01224de923","Type":"ContainerDied","Data":"42902be1acdfb6ca8eed153875c5e1d3edc963d481f088a2862b00567be86dc2"} Sep 17 18:30:33.546484 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.546372 2564 scope.go:117] "RemoveContainer" containerID="1c8445718c28d1f20cff28599d726d19e92176982857f2478500590e89ba3f6f" Sep 17 18:30:33.558976 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.558960 2564 scope.go:117] "RemoveContainer" containerID="6197b2d4f63aa2cedbbfef38a88669555eddea014605a546477d7d7aa025e5fb" Sep 17 18:30:33.567738 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.567722 2564 scope.go:117] "RemoveContainer" containerID="1c8445718c28d1f20cff28599d726d19e92176982857f2478500590e89ba3f6f" Sep 17 18:30:33.568014 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:33.567997 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8445718c28d1f20cff28599d726d19e92176982857f2478500590e89ba3f6f\": container with ID starting with 1c8445718c28d1f20cff28599d726d19e92176982857f2478500590e89ba3f6f not found: ID does not exist" containerID="1c8445718c28d1f20cff28599d726d19e92176982857f2478500590e89ba3f6f" Sep 17 18:30:33.568089 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.568020 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8445718c28d1f20cff28599d726d19e92176982857f2478500590e89ba3f6f"} err="failed to get container status \"1c8445718c28d1f20cff28599d726d19e92176982857f2478500590e89ba3f6f\": rpc error: code = NotFound desc = could not find container \"1c8445718c28d1f20cff28599d726d19e92176982857f2478500590e89ba3f6f\": container with ID starting with 1c8445718c28d1f20cff28599d726d19e92176982857f2478500590e89ba3f6f not found: ID does not exist" Sep 17 18:30:33.568089 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.568041 2564 scope.go:117] "RemoveContainer" containerID="6197b2d4f63aa2cedbbfef38a88669555eddea014605a546477d7d7aa025e5fb" Sep 17 18:30:33.568326 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:33.568305 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6197b2d4f63aa2cedbbfef38a88669555eddea014605a546477d7d7aa025e5fb\": container with ID starting with 6197b2d4f63aa2cedbbfef38a88669555eddea014605a546477d7d7aa025e5fb not found: ID does not exist" containerID="6197b2d4f63aa2cedbbfef38a88669555eddea014605a546477d7d7aa025e5fb" Sep 17 18:30:33.568371 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.568336 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6197b2d4f63aa2cedbbfef38a88669555eddea014605a546477d7d7aa025e5fb"} err="failed to get container status \"6197b2d4f63aa2cedbbfef38a88669555eddea014605a546477d7d7aa025e5fb\": rpc error: code = NotFound desc = could not find container \"6197b2d4f63aa2cedbbfef38a88669555eddea014605a546477d7d7aa025e5fb\": container with ID starting with 6197b2d4f63aa2cedbbfef38a88669555eddea014605a546477d7d7aa025e5fb not found: ID does not exist" Sep 17 18:30:33.576229 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.576210 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb"] Sep 17 18:30:33.579770 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.579738 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b06908-openshift-default-bd57dd984-6wddb"] Sep 17 18:30:33.603049 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.603031 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:30:33.788644 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.788614 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-dshm\") pod \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " Sep 17 18:30:33.788814 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.788661 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-tmp-dir\") pod \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " Sep 17 18:30:33.788814 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.788715 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjp6s\" (UniqueName: \"kubernetes.io/projected/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-kube-api-access-mjp6s\") pod \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " Sep 17 18:30:33.788814 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.788790 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-model-cache\") pod \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " Sep 17 18:30:33.788988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.788865 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-home\") pod \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " Sep 17 18:30:33.788988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.788892 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-tls-certs\") pod \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\" (UID: \"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2\") " Sep 17 18:30:33.788988 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.788943 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b14b32c9-c9df-4054-a0bd-6f9a9dc676e2" (UID: "b14b32c9-c9df-4054-a0bd-6f9a9dc676e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:33.789140 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.789100 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "b14b32c9-c9df-4054-a0bd-6f9a9dc676e2" (UID: "b14b32c9-c9df-4054-a0bd-6f9a9dc676e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:33.789140 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.789115 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-home" (OuterVolumeSpecName: "home") pod "b14b32c9-c9df-4054-a0bd-6f9a9dc676e2" (UID: "b14b32c9-c9df-4054-a0bd-6f9a9dc676e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:33.789298 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.789272 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-model-cache\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:33.789298 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.789294 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-home\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:33.789298 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.789303 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-tmp-dir\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:33.791235 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.791208 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-dshm" (OuterVolumeSpecName: "dshm") pod "b14b32c9-c9df-4054-a0bd-6f9a9dc676e2" (UID: "b14b32c9-c9df-4054-a0bd-6f9a9dc676e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:33.791380 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.791260 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-kube-api-access-mjp6s" (OuterVolumeSpecName: "kube-api-access-mjp6s") pod "b14b32c9-c9df-4054-a0bd-6f9a9dc676e2" (UID: "b14b32c9-c9df-4054-a0bd-6f9a9dc676e2"). InnerVolumeSpecName "kube-api-access-mjp6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:33.791380 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.791313 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b14b32c9-c9df-4054-a0bd-6f9a9dc676e2" (UID: "b14b32c9-c9df-4054-a0bd-6f9a9dc676e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:30:33.889940 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.889872 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-tls-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:33.889940 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.889895 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-dshm\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:33.889940 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:33.889904 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjp6s\" (UniqueName: \"kubernetes.io/projected/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2-kube-api-access-mjp6s\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:34.554166 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:34.554134 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" Sep 17 18:30:34.554624 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:34.554160 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb" event={"ID":"b14b32c9-c9df-4054-a0bd-6f9a9dc676e2","Type":"ContainerDied","Data":"32b38c674bd98a7607dfbefc01e1ef3bde65529748c91eb9c32ebaa0ec9d3a25"} Sep 17 18:30:34.554624 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:34.554213 2564 scope.go:117] "RemoveContainer" containerID="296c94ea45793c73071c86bbeb2e7efef09a8a3b55ec8f7c514aa1aafe4788b2" Sep 17 18:30:34.577158 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:34.577128 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb"] Sep 17 18:30:34.580653 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:34.580631 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/distinct-w0-5d8053-kserve-865984d9f8-5t5hb"] Sep 17 18:30:35.453284 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:35.453242 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b14b32c9-c9df-4054-a0bd-6f9a9dc676e2" path="/var/lib/kubelet/pods/b14b32c9-c9df-4054-a0bd-6f9a9dc676e2/volumes" Sep 17 18:30:35.453767 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:35.453745 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" path="/var/lib/kubelet/pods/e21c0cc6-1c7b-4d20-8fa1-2f01224de923/volumes" Sep 17 18:30:38.761230 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:38.761187 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj"] Sep 17 18:30:38.763844 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:38.763822 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1a9245-29827830-95wfj"] Sep 17 18:30:38.802992 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:38.802962 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7"] Sep 17 18:30:38.803279 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:38.803247 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" podUID="a253f7c8-24c5-4caf-b6d0-98127fd0d902" containerName="payload-processing" containerID="cri-o://959ba43e07db7d501159ab7ac4d60fdbb2808587a4d3baba5661ee2b3e93d1ac" gracePeriod=30 Sep 17 18:30:38.807770 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:38.807745 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt"] Sep 17 18:30:38.808079 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:38.808053 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" podUID="685be49a-b0ef-4c07-9347-a29acabe48d3" containerName="payload-pre-processing" containerID="cri-o://3ec03b095cd91429e746834df134376400ae181d666372a1fba2b5689c9347c2" gracePeriod=30 Sep 17 18:30:39.094762 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.094736 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" Sep 17 18:30:39.097499 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.097481 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" Sep 17 18:30:39.134520 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.133065 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685be49a-b0ef-4c07-9347-a29acabe48d3-plugins-config-volume\") pod \"685be49a-b0ef-4c07-9347-a29acabe48d3\" (UID: \"685be49a-b0ef-4c07-9347-a29acabe48d3\") " Sep 17 18:30:39.134520 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.133128 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a253f7c8-24c5-4caf-b6d0-98127fd0d902-plugins-config-volume\") pod \"a253f7c8-24c5-4caf-b6d0-98127fd0d902\" (UID: \"a253f7c8-24c5-4caf-b6d0-98127fd0d902\") " Sep 17 18:30:39.134520 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.133156 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqcbp\" (UniqueName: \"kubernetes.io/projected/a253f7c8-24c5-4caf-b6d0-98127fd0d902-kube-api-access-kqcbp\") pod \"a253f7c8-24c5-4caf-b6d0-98127fd0d902\" (UID: \"a253f7c8-24c5-4caf-b6d0-98127fd0d902\") " Sep 17 18:30:39.134520 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.133221 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx8pd\" (UniqueName: \"kubernetes.io/projected/685be49a-b0ef-4c07-9347-a29acabe48d3-kube-api-access-tx8pd\") pod \"685be49a-b0ef-4c07-9347-a29acabe48d3\" (UID: \"685be49a-b0ef-4c07-9347-a29acabe48d3\") " Sep 17 18:30:39.135497 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.135453 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685be49a-b0ef-4c07-9347-a29acabe48d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "685be49a-b0ef-4c07-9347-a29acabe48d3" (UID: "685be49a-b0ef-4c07-9347-a29acabe48d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:39.136274 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.136178 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a253f7c8-24c5-4caf-b6d0-98127fd0d902-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a253f7c8-24c5-4caf-b6d0-98127fd0d902" (UID: "a253f7c8-24c5-4caf-b6d0-98127fd0d902"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:39.139453 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.139416 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685be49a-b0ef-4c07-9347-a29acabe48d3-kube-api-access-tx8pd" (OuterVolumeSpecName: "kube-api-access-tx8pd") pod "685be49a-b0ef-4c07-9347-a29acabe48d3" (UID: "685be49a-b0ef-4c07-9347-a29acabe48d3"). InnerVolumeSpecName "kube-api-access-tx8pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:39.140103 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.140082 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a253f7c8-24c5-4caf-b6d0-98127fd0d902-kube-api-access-kqcbp" (OuterVolumeSpecName: "kube-api-access-kqcbp") pod "a253f7c8-24c5-4caf-b6d0-98127fd0d902" (UID: "a253f7c8-24c5-4caf-b6d0-98127fd0d902"). InnerVolumeSpecName "kube-api-access-kqcbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:39.234397 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.234357 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685be49a-b0ef-4c07-9347-a29acabe48d3-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:39.234397 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.234391 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a253f7c8-24c5-4caf-b6d0-98127fd0d902-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:39.234397 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.234400 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqcbp\" (UniqueName: \"kubernetes.io/projected/a253f7c8-24c5-4caf-b6d0-98127fd0d902-kube-api-access-kqcbp\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:39.234397 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.234409 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx8pd\" (UniqueName: \"kubernetes.io/projected/685be49a-b0ef-4c07-9347-a29acabe48d3-kube-api-access-tx8pd\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:39.451238 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.451207 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb50ff34-9c97-4f32-870a-a120e2bd7681" path="/var/lib/kubelet/pods/eb50ff34-9c97-4f32-870a-a120e2bd7681/volumes" Sep 17 18:30:39.589814 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.589780 2564 generic.go:356] "Generic (PLEG): container finished" podID="a253f7c8-24c5-4caf-b6d0-98127fd0d902" containerID="959ba43e07db7d501159ab7ac4d60fdbb2808587a4d3baba5661ee2b3e93d1ac" exitCode=0 Sep 17 18:30:39.590031 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.589827 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" Sep 17 18:30:39.590031 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.589870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" event={"ID":"a253f7c8-24c5-4caf-b6d0-98127fd0d902","Type":"ContainerDied","Data":"959ba43e07db7d501159ab7ac4d60fdbb2808587a4d3baba5661ee2b3e93d1ac"} Sep 17 18:30:39.590031 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.589926 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7" event={"ID":"a253f7c8-24c5-4caf-b6d0-98127fd0d902","Type":"ContainerDied","Data":"8b419f314c048555b4652644ea59b94dae679790e8fd745cf5127b8c09c45f7c"} Sep 17 18:30:39.590031 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.589954 2564 scope.go:117] "RemoveContainer" containerID="959ba43e07db7d501159ab7ac4d60fdbb2808587a4d3baba5661ee2b3e93d1ac" Sep 17 18:30:39.591387 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.591364 2564 generic.go:356] "Generic (PLEG): container finished" podID="685be49a-b0ef-4c07-9347-a29acabe48d3" containerID="3ec03b095cd91429e746834df134376400ae181d666372a1fba2b5689c9347c2" exitCode=0 Sep 17 18:30:39.591490 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.591417 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" Sep 17 18:30:39.591490 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.591458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" event={"ID":"685be49a-b0ef-4c07-9347-a29acabe48d3","Type":"ContainerDied","Data":"3ec03b095cd91429e746834df134376400ae181d666372a1fba2b5689c9347c2"} Sep 17 18:30:39.591645 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.591521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt" event={"ID":"685be49a-b0ef-4c07-9347-a29acabe48d3","Type":"ContainerDied","Data":"8f44bce38d88b4fd5030ab40a57c9a7f841f3a8db7a0fcbc90a613d33c48a34a"} Sep 17 18:30:39.599962 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.599925 2564 scope.go:117] "RemoveContainer" containerID="959ba43e07db7d501159ab7ac4d60fdbb2808587a4d3baba5661ee2b3e93d1ac" Sep 17 18:30:39.600222 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:39.600201 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959ba43e07db7d501159ab7ac4d60fdbb2808587a4d3baba5661ee2b3e93d1ac\": container with ID starting with 959ba43e07db7d501159ab7ac4d60fdbb2808587a4d3baba5661ee2b3e93d1ac not found: ID does not exist" containerID="959ba43e07db7d501159ab7ac4d60fdbb2808587a4d3baba5661ee2b3e93d1ac" Sep 17 18:30:39.600287 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.600230 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959ba43e07db7d501159ab7ac4d60fdbb2808587a4d3baba5661ee2b3e93d1ac"} err="failed to get container status \"959ba43e07db7d501159ab7ac4d60fdbb2808587a4d3baba5661ee2b3e93d1ac\": rpc error: code = NotFound desc = could not find container \"959ba43e07db7d501159ab7ac4d60fdbb2808587a4d3baba5661ee2b3e93d1ac\": container with ID starting with 959ba43e07db7d501159ab7ac4d60fdbb2808587a4d3baba5661ee2b3e93d1ac not found: ID does not exist" Sep 17 18:30:39.600287 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.600246 2564 scope.go:117] "RemoveContainer" containerID="3ec03b095cd91429e746834df134376400ae181d666372a1fba2b5689c9347c2" Sep 17 18:30:39.609242 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.609225 2564 scope.go:117] "RemoveContainer" containerID="3ec03b095cd91429e746834df134376400ae181d666372a1fba2b5689c9347c2" Sep 17 18:30:39.609547 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:39.609504 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec03b095cd91429e746834df134376400ae181d666372a1fba2b5689c9347c2\": container with ID starting with 3ec03b095cd91429e746834df134376400ae181d666372a1fba2b5689c9347c2 not found: ID does not exist" containerID="3ec03b095cd91429e746834df134376400ae181d666372a1fba2b5689c9347c2" Sep 17 18:30:39.609627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.609581 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec03b095cd91429e746834df134376400ae181d666372a1fba2b5689c9347c2"} err="failed to get container status \"3ec03b095cd91429e746834df134376400ae181d666372a1fba2b5689c9347c2\": rpc error: code = NotFound desc = could not find container \"3ec03b095cd91429e746834df134376400ae181d666372a1fba2b5689c9347c2\": container with ID starting with 3ec03b095cd91429e746834df134376400ae181d666372a1fba2b5689c9347c2 not found: ID does not exist" Sep 17 18:30:39.610063 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.610046 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7"] Sep 17 18:30:39.615658 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.615635 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1a9245-5f57cc69c-kdgk7"] Sep 17 18:30:39.626960 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.626939 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt"] Sep 17 18:30:39.631089 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.631070 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1a9245-57c455d64b-f56tt"] Sep 17 18:30:39.915110 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.915074 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz"] Sep 17 18:30:39.915589 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:39.915337 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerName="istio-proxy" containerID="cri-o://ff4b1a9932bcf42b5bf3c64af2f05fb0f06690045d4cd9a5634d367b90d51161" gracePeriod=30 Sep 17 18:30:41.457192 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:41.457157 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="685be49a-b0ef-4c07-9347-a29acabe48d3" path="/var/lib/kubelet/pods/685be49a-b0ef-4c07-9347-a29acabe48d3/volumes" Sep 17 18:30:41.457722 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:41.457514 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a253f7c8-24c5-4caf-b6d0-98127fd0d902" path="/var/lib/kubelet/pods/a253f7c8-24c5-4caf-b6d0-98127fd0d902/volumes" Sep 17 18:30:45.165614 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.165587 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:30:45.184288 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184232 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-credential-socket\") pod \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " Sep 17 18:30:45.184288 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184257 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-workload-socket\") pod \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " Sep 17 18:30:45.184288 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184286 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-envoy\") pod \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " Sep 17 18:30:45.184576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184317 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-token\") pod \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " Sep 17 18:30:45.184576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184356 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-data\") pod \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " Sep 17 18:30:45.184576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184381 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istiod-ca-cert\") pod \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " Sep 17 18:30:45.184576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184463 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-workload-certs\") pod \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " Sep 17 18:30:45.184576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184484 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92979558-f5c7-46c0-9eb2-3cdaf8edb19d" (UID: "92979558-f5c7-46c0-9eb2-3cdaf8edb19d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:45.184576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184507 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97xmn\" (UniqueName: \"kubernetes.io/projected/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-kube-api-access-97xmn\") pod \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " Sep 17 18:30:45.184576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184509 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92979558-f5c7-46c0-9eb2-3cdaf8edb19d" (UID: "92979558-f5c7-46c0-9eb2-3cdaf8edb19d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:45.184576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184576 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-podinfo\") pod \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\" (UID: \"92979558-f5c7-46c0-9eb2-3cdaf8edb19d\") " Sep 17 18:30:45.185180 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184672 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92979558-f5c7-46c0-9eb2-3cdaf8edb19d" (UID: "92979558-f5c7-46c0-9eb2-3cdaf8edb19d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:45.185180 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184694 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-data" (OuterVolumeSpecName: "istio-data") pod "92979558-f5c7-46c0-9eb2-3cdaf8edb19d" (UID: "92979558-f5c7-46c0-9eb2-3cdaf8edb19d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:45.185180 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184836 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92979558-f5c7-46c0-9eb2-3cdaf8edb19d" (UID: "92979558-f5c7-46c0-9eb2-3cdaf8edb19d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:45.185180 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184971 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:45.185180 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.184992 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:45.185180 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.185008 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:45.185180 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.185020 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:45.185180 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.185032 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:45.186751 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.186719 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92979558-f5c7-46c0-9eb2-3cdaf8edb19d" (UID: "92979558-f5c7-46c0-9eb2-3cdaf8edb19d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:45.186857 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.186745 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-token" (OuterVolumeSpecName: "istio-token") pod "92979558-f5c7-46c0-9eb2-3cdaf8edb19d" (UID: "92979558-f5c7-46c0-9eb2-3cdaf8edb19d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:45.186857 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.186751 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-kube-api-access-97xmn" (OuterVolumeSpecName: "kube-api-access-97xmn") pod "92979558-f5c7-46c0-9eb2-3cdaf8edb19d" (UID: "92979558-f5c7-46c0-9eb2-3cdaf8edb19d"). InnerVolumeSpecName "kube-api-access-97xmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:45.186857 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.186795 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92979558-f5c7-46c0-9eb2-3cdaf8edb19d" (UID: "92979558-f5c7-46c0-9eb2-3cdaf8edb19d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:30:45.285928 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.285898 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97xmn\" (UniqueName: \"kubernetes.io/projected/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-kube-api-access-97xmn\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:45.285928 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.285926 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:45.286071 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.285936 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:45.286071 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.285944 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92979558-f5c7-46c0-9eb2-3cdaf8edb19d-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:45.640279 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.640239 2564 generic.go:356] "Generic (PLEG): container finished" podID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerID="ff4b1a9932bcf42b5bf3c64af2f05fb0f06690045d4cd9a5634d367b90d51161" exitCode=0 Sep 17 18:30:45.640426 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.640305 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" Sep 17 18:30:45.640426 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.640330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" event={"ID":"92979558-f5c7-46c0-9eb2-3cdaf8edb19d","Type":"ContainerDied","Data":"ff4b1a9932bcf42b5bf3c64af2f05fb0f06690045d4cd9a5634d367b90d51161"} Sep 17 18:30:45.640578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.640430 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz" event={"ID":"92979558-f5c7-46c0-9eb2-3cdaf8edb19d","Type":"ContainerDied","Data":"7fa7c882088a9576704cdaa28e1c96faf38eb317493b0a523fe4e27e491a3ff5"} Sep 17 18:30:45.640578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.640465 2564 scope.go:117] "RemoveContainer" containerID="ff4b1a9932bcf42b5bf3c64af2f05fb0f06690045d4cd9a5634d367b90d51161" Sep 17 18:30:45.650499 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.650480 2564 scope.go:117] "RemoveContainer" containerID="962e5636cdce719e8589716aec68cbb9e6167d491d16ef0bfd1898fe8dd698b6" Sep 17 18:30:45.661694 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.661670 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz"] Sep 17 18:30:45.661917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.661897 2564 scope.go:117] "RemoveContainer" containerID="ff4b1a9932bcf42b5bf3c64af2f05fb0f06690045d4cd9a5634d367b90d51161" Sep 17 18:30:45.662978 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:45.662951 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4b1a9932bcf42b5bf3c64af2f05fb0f06690045d4cd9a5634d367b90d51161\": container with ID starting with ff4b1a9932bcf42b5bf3c64af2f05fb0f06690045d4cd9a5634d367b90d51161 not found: ID does not exist" containerID="ff4b1a9932bcf42b5bf3c64af2f05fb0f06690045d4cd9a5634d367b90d51161" Sep 17 18:30:45.663069 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.662989 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4b1a9932bcf42b5bf3c64af2f05fb0f06690045d4cd9a5634d367b90d51161"} err="failed to get container status \"ff4b1a9932bcf42b5bf3c64af2f05fb0f06690045d4cd9a5634d367b90d51161\": rpc error: code = NotFound desc = could not find container \"ff4b1a9932bcf42b5bf3c64af2f05fb0f06690045d4cd9a5634d367b90d51161\": container with ID starting with ff4b1a9932bcf42b5bf3c64af2f05fb0f06690045d4cd9a5634d367b90d51161 not found: ID does not exist" Sep 17 18:30:45.663069 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.663011 2564 scope.go:117] "RemoveContainer" containerID="962e5636cdce719e8589716aec68cbb9e6167d491d16ef0bfd1898fe8dd698b6" Sep 17 18:30:45.663346 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:45.663326 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962e5636cdce719e8589716aec68cbb9e6167d491d16ef0bfd1898fe8dd698b6\": container with ID starting with 962e5636cdce719e8589716aec68cbb9e6167d491d16ef0bfd1898fe8dd698b6 not found: ID does not exist" containerID="962e5636cdce719e8589716aec68cbb9e6167d491d16ef0bfd1898fe8dd698b6" Sep 17 18:30:45.663402 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.663351 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962e5636cdce719e8589716aec68cbb9e6167d491d16ef0bfd1898fe8dd698b6"} err="failed to get container status \"962e5636cdce719e8589716aec68cbb9e6167d491d16ef0bfd1898fe8dd698b6\": rpc error: code = NotFound desc = could not find container \"962e5636cdce719e8589716aec68cbb9e6167d491d16ef0bfd1898fe8dd698b6\": container with ID starting with 962e5636cdce719e8589716aec68cbb9e6167d491d16ef0bfd1898fe8dd698b6 not found: ID does not exist" Sep 17 18:30:45.665128 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:45.665110 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5d8053-openshift-default-9b5dcddf8-qm6dz"] Sep 17 18:30:46.033203 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.033173 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847"] Sep 17 18:30:46.033420 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.033397 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerName="istio-proxy" containerID="cri-o://3afefff311469a3f2943b436a202c3d6ed6d79509aec6da7174ea2049ab88e4d" gracePeriod=30 Sep 17 18:30:46.386097 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386068 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5"] Sep 17 18:30:46.386489 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386474 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerName="istio-proxy" Sep 17 18:30:46.386569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386490 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerName="istio-proxy" Sep 17 18:30:46.386569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386502 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerName="istio-proxy" Sep 17 18:30:46.386569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386507 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerName="istio-proxy" Sep 17 18:30:46.386569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386516 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12795c6d-6d9c-4ce8-a6be-decbb9f9af6a" containerName="cleanup" Sep 17 18:30:46.386569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386522 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="12795c6d-6d9c-4ce8-a6be-decbb9f9af6a" containerName="cleanup" Sep 17 18:30:46.386569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386542 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" Sep 17 18:30:46.386569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386548 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" Sep 17 18:30:46.386569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386555 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d6d36a-bc35-4381-a8bd-9badc7e3d03a" containerName="payload-pre-processing" Sep 17 18:30:46.386569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386561 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d6d36a-bc35-4381-a8bd-9badc7e3d03a" containerName="payload-pre-processing" Sep 17 18:30:46.386569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386568 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b14b32c9-c9df-4054-a0bd-6f9a9dc676e2" containerName="main" Sep 17 18:30:46.386569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386575 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b14b32c9-c9df-4054-a0bd-6f9a9dc676e2" containerName="main" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386582 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="685be49a-b0ef-4c07-9347-a29acabe48d3" containerName="payload-pre-processing" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386587 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="685be49a-b0ef-4c07-9347-a29acabe48d3" containerName="payload-pre-processing" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386594 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb50ff34-9c97-4f32-870a-a120e2bd7681" containerName="cleanup" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386598 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb50ff34-9c97-4f32-870a-a120e2bd7681" containerName="cleanup" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386611 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c73ee8-66dd-4f6e-9cf9-87696e8cc69b" containerName="payload-pre-processing" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386616 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c73ee8-66dd-4f6e-9cf9-87696e8cc69b" containerName="payload-pre-processing" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386624 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a253f7c8-24c5-4caf-b6d0-98127fd0d902" containerName="payload-processing" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386629 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a253f7c8-24c5-4caf-b6d0-98127fd0d902" containerName="payload-processing" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386641 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bde362a-db00-456e-954d-263597ad8c1e" containerName="payload-processing" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386646 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bde362a-db00-456e-954d-263597ad8c1e" containerName="payload-processing" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386650 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b55eacb-6168-4165-8001-c2827eeb1efa" containerName="cleanup" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386655 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b55eacb-6168-4165-8001-c2827eeb1efa" containerName="cleanup" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386715 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c73ee8-66dd-4f6e-9cf9-87696e8cc69b" containerName="payload-pre-processing" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386723 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386728 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="12795c6d-6d9c-4ce8-a6be-decbb9f9af6a" containerName="cleanup" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386734 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerName="istio-proxy" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386738 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb50ff34-9c97-4f32-870a-a120e2bd7681" containerName="cleanup" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386745 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b55eacb-6168-4165-8001-c2827eeb1efa" containerName="cleanup" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386750 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="685be49a-b0ef-4c07-9347-a29acabe48d3" containerName="payload-pre-processing" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386756 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerName="istio-proxy" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386761 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b14b32c9-c9df-4054-a0bd-6f9a9dc676e2" containerName="main" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386766 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d6d36a-bc35-4381-a8bd-9badc7e3d03a" containerName="payload-pre-processing" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386772 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bde362a-db00-456e-954d-263597ad8c1e" containerName="payload-processing" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386778 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerName="istio-proxy" Sep 17 18:30:46.386906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.386786 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a253f7c8-24c5-4caf-b6d0-98127fd0d902" containerName="payload-processing" Sep 17 18:30:46.391515 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.391498 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5" Sep 17 18:30:46.394068 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.394040 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 18:30:46.394188 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.394095 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\"" Sep 17 18:30:46.399333 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.399311 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5"] Sep 17 18:30:46.496121 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.496091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4a9503-62f8-43d5-b238-a6c536c8846a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5\" (UID: \"5d4a9503-62f8-43d5-b238-a6c536c8846a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5" Sep 17 18:30:46.596923 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.596900 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4a9503-62f8-43d5-b238-a6c536c8846a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5\" (UID: \"5d4a9503-62f8-43d5-b238-a6c536c8846a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5" Sep 17 18:30:46.597448 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.597431 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4a9503-62f8-43d5-b238-a6c536c8846a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5\" (UID: \"5d4a9503-62f8-43d5-b238-a6c536c8846a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5" Sep 17 18:30:46.703332 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.703268 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5" Sep 17 18:30:46.976926 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:46.976899 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5"] Sep 17 18:30:46.978226 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:30:46.978199 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4a9503_62f8_43d5_b238_a6c536c8846a.slice/crio-8743e0dab354915b59e4e2bbe517d27764d8c50e0ee749901d12eab60fd48b62 WatchSource:0}: Error finding container 8743e0dab354915b59e4e2bbe517d27764d8c50e0ee749901d12eab60fd48b62: Status 404 returned error can't find the container with id 8743e0dab354915b59e4e2bbe517d27764d8c50e0ee749901d12eab60fd48b62 Sep 17 18:30:47.452261 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:47.452233 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" path="/var/lib/kubelet/pods/92979558-f5c7-46c0-9eb2-3cdaf8edb19d/volumes" Sep 17 18:30:47.654614 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:47.654585 2564 generic.go:356] "Generic (PLEG): container finished" podID="5d4a9503-62f8-43d5-b238-a6c536c8846a" containerID="86277945eaa5cc539db670b9e527883ce501d96efca4911672900d53bddf5694" exitCode=0 Sep 17 18:30:47.654719 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:47.654670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5" event={"ID":"5d4a9503-62f8-43d5-b238-a6c536c8846a","Type":"ContainerDied","Data":"86277945eaa5cc539db670b9e527883ce501d96efca4911672900d53bddf5694"} Sep 17 18:30:47.654772 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:47.654717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5" event={"ID":"5d4a9503-62f8-43d5-b238-a6c536c8846a","Type":"ContainerStarted","Data":"8743e0dab354915b59e4e2bbe517d27764d8c50e0ee749901d12eab60fd48b62"} Sep 17 18:30:48.791339 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:48.791318 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5" Sep 17 18:30:48.818899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:48.818876 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4a9503-62f8-43d5-b238-a6c536c8846a-maas-api-service-ca\") pod \"5d4a9503-62f8-43d5-b238-a6c536c8846a\" (UID: \"5d4a9503-62f8-43d5-b238-a6c536c8846a\") " Sep 17 18:30:48.819278 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:48.819252 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4a9503-62f8-43d5-b238-a6c536c8846a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d4a9503-62f8-43d5-b238-a6c536c8846a" (UID: "5d4a9503-62f8-43d5-b238-a6c536c8846a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:48.920031 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:48.919972 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4a9503-62f8-43d5-b238-a6c536c8846a-maas-api-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:49.668827 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:49.668789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5" event={"ID":"5d4a9503-62f8-43d5-b238-a6c536c8846a","Type":"ContainerDied","Data":"8743e0dab354915b59e4e2bbe517d27764d8c50e0ee749901d12eab60fd48b62"} Sep 17 18:30:49.668956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:49.668856 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8743e0dab354915b59e4e2bbe517d27764d8c50e0ee749901d12eab60fd48b62" Sep 17 18:30:49.668956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:49.668801 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5" Sep 17 18:30:51.284790 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.284768 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:30:51.339785 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.339756 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-credential-socket\") pod \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " Sep 17 18:30:51.339917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.339801 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f22vm\" (UniqueName: \"kubernetes.io/projected/c637b203-8705-4b9b-b5b5-9fa6728be0b8-kube-api-access-f22vm\") pod \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " Sep 17 18:30:51.339917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.339863 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-workload-socket\") pod \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " Sep 17 18:30:51.339917 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.339887 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-token\") pod \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " Sep 17 18:30:51.340083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.339915 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istiod-ca-cert\") pod \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " Sep 17 18:30:51.340083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.339950 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-workload-certs\") pod \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " Sep 17 18:30:51.340083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.340009 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-envoy\") pod \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " Sep 17 18:30:51.340083 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.340052 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-podinfo\") pod \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " Sep 17 18:30:51.340283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.340088 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-data\") pod \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\" (UID: \"c637b203-8705-4b9b-b5b5-9fa6728be0b8\") " Sep 17 18:30:51.340283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.340002 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c637b203-8705-4b9b-b5b5-9fa6728be0b8" (UID: "c637b203-8705-4b9b-b5b5-9fa6728be0b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:51.340283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.340191 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c637b203-8705-4b9b-b5b5-9fa6728be0b8" (UID: "c637b203-8705-4b9b-b5b5-9fa6728be0b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:51.340283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.340226 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c637b203-8705-4b9b-b5b5-9fa6728be0b8" (UID: "c637b203-8705-4b9b-b5b5-9fa6728be0b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:51.340493 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.340456 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "c637b203-8705-4b9b-b5b5-9fa6728be0b8" (UID: "c637b203-8705-4b9b-b5b5-9fa6728be0b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:51.340493 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.340481 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:51.340639 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.340503 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:51.340639 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.340518 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:51.340743 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.340666 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c637b203-8705-4b9b-b5b5-9fa6728be0b8" (UID: "c637b203-8705-4b9b-b5b5-9fa6728be0b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:51.342326 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.342302 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c637b203-8705-4b9b-b5b5-9fa6728be0b8" (UID: "c637b203-8705-4b9b-b5b5-9fa6728be0b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:51.342432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.342386 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c637b203-8705-4b9b-b5b5-9fa6728be0b8" (UID: "c637b203-8705-4b9b-b5b5-9fa6728be0b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:30:51.342674 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.342649 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "c637b203-8705-4b9b-b5b5-9fa6728be0b8" (UID: "c637b203-8705-4b9b-b5b5-9fa6728be0b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:51.342819 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.342797 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c637b203-8705-4b9b-b5b5-9fa6728be0b8-kube-api-access-f22vm" (OuterVolumeSpecName: "kube-api-access-f22vm") pod "c637b203-8705-4b9b-b5b5-9fa6728be0b8" (UID: "c637b203-8705-4b9b-b5b5-9fa6728be0b8"). InnerVolumeSpecName "kube-api-access-f22vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:51.441227 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.441206 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:51.441227 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.441224 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:51.441346 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.441235 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:51.441346 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.441244 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:51.441346 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.441251 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c637b203-8705-4b9b-b5b5-9fa6728be0b8-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:51.441346 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.441260 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f22vm\" (UniqueName: \"kubernetes.io/projected/c637b203-8705-4b9b-b5b5-9fa6728be0b8-kube-api-access-f22vm\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:51.682777 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.682712 2564 generic.go:356] "Generic (PLEG): container finished" podID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerID="3afefff311469a3f2943b436a202c3d6ed6d79509aec6da7174ea2049ab88e4d" exitCode=0 Sep 17 18:30:51.682777 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.682770 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" Sep 17 18:30:51.682910 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.682796 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" event={"ID":"c637b203-8705-4b9b-b5b5-9fa6728be0b8","Type":"ContainerDied","Data":"3afefff311469a3f2943b436a202c3d6ed6d79509aec6da7174ea2049ab88e4d"} Sep 17 18:30:51.682910 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.682847 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847" event={"ID":"c637b203-8705-4b9b-b5b5-9fa6728be0b8","Type":"ContainerDied","Data":"1daae948eb93d88f71937f73d318fa8042d4bad0536b9421256634b9ea1f1058"} Sep 17 18:30:51.682910 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.682869 2564 scope.go:117] "RemoveContainer" containerID="3afefff311469a3f2943b436a202c3d6ed6d79509aec6da7174ea2049ab88e4d" Sep 17 18:30:51.692403 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.692385 2564 scope.go:117] "RemoveContainer" containerID="354906a9559eaac75d70493c89d4ca901bf459abc11ce43fad7713bf6ce20572" Sep 17 18:30:51.701272 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.701253 2564 scope.go:117] "RemoveContainer" containerID="3afefff311469a3f2943b436a202c3d6ed6d79509aec6da7174ea2049ab88e4d" Sep 17 18:30:51.701498 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:51.701483 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afefff311469a3f2943b436a202c3d6ed6d79509aec6da7174ea2049ab88e4d\": container with ID starting with 3afefff311469a3f2943b436a202c3d6ed6d79509aec6da7174ea2049ab88e4d not found: ID does not exist" containerID="3afefff311469a3f2943b436a202c3d6ed6d79509aec6da7174ea2049ab88e4d" Sep 17 18:30:51.701585 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.701506 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afefff311469a3f2943b436a202c3d6ed6d79509aec6da7174ea2049ab88e4d"} err="failed to get container status \"3afefff311469a3f2943b436a202c3d6ed6d79509aec6da7174ea2049ab88e4d\": rpc error: code = NotFound desc = could not find container \"3afefff311469a3f2943b436a202c3d6ed6d79509aec6da7174ea2049ab88e4d\": container with ID starting with 3afefff311469a3f2943b436a202c3d6ed6d79509aec6da7174ea2049ab88e4d not found: ID does not exist" Sep 17 18:30:51.701585 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.701522 2564 scope.go:117] "RemoveContainer" containerID="354906a9559eaac75d70493c89d4ca901bf459abc11ce43fad7713bf6ce20572" Sep 17 18:30:51.701846 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:51.701830 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354906a9559eaac75d70493c89d4ca901bf459abc11ce43fad7713bf6ce20572\": container with ID starting with 354906a9559eaac75d70493c89d4ca901bf459abc11ce43fad7713bf6ce20572 not found: ID does not exist" containerID="354906a9559eaac75d70493c89d4ca901bf459abc11ce43fad7713bf6ce20572" Sep 17 18:30:51.701911 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.701851 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354906a9559eaac75d70493c89d4ca901bf459abc11ce43fad7713bf6ce20572"} err="failed to get container status \"354906a9559eaac75d70493c89d4ca901bf459abc11ce43fad7713bf6ce20572\": rpc error: code = NotFound desc = could not find container \"354906a9559eaac75d70493c89d4ca901bf459abc11ce43fad7713bf6ce20572\": container with ID starting with 354906a9559eaac75d70493c89d4ca901bf459abc11ce43fad7713bf6ce20572 not found: ID does not exist" Sep 17 18:30:51.702196 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.702179 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847"] Sep 17 18:30:51.706650 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:51.706628 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1a9245-openshift-default-78b847994-nl847"] Sep 17 18:30:53.451828 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:53.451795 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" path="/var/lib/kubelet/pods/c637b203-8705-4b9b-b5b5-9fa6728be0b8/volumes" Sep 17 18:30:57.075578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.075547 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv"] Sep 17 18:30:57.075980 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.075776 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" podUID="e62d8345-fe5a-49c5-9360-60f50fca71eb" containerName="payload-processing" containerID="cri-o://85bec18417df1721b68606dd27d297dc0c752594c8a1eee50657afaf9f90b283" gracePeriod=30 Sep 17 18:30:57.084226 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.084052 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87"] Sep 17 18:30:57.084412 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.084371 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" podUID="7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da" containerName="payload-pre-processing" containerID="cri-o://dcb5097189c8306b0e8c99abc577850d532879f7e7b7d23d51b52b101f0dd1a8" gracePeriod=30 Sep 17 18:30:57.266869 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.266808 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5"] Sep 17 18:30:57.272131 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.272073 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-03270e-7mwl5"] Sep 17 18:30:57.352404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.352377 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" Sep 17 18:30:57.374890 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.374866 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" Sep 17 18:30:57.391744 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.391679 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtwpg\" (UniqueName: \"kubernetes.io/projected/e62d8345-fe5a-49c5-9360-60f50fca71eb-kube-api-access-qtwpg\") pod \"e62d8345-fe5a-49c5-9360-60f50fca71eb\" (UID: \"e62d8345-fe5a-49c5-9360-60f50fca71eb\") " Sep 17 18:30:57.391875 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.391798 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62d8345-fe5a-49c5-9360-60f50fca71eb-plugins-config-volume\") pod \"e62d8345-fe5a-49c5-9360-60f50fca71eb\" (UID: \"e62d8345-fe5a-49c5-9360-60f50fca71eb\") " Sep 17 18:30:57.392664 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.392606 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62d8345-fe5a-49c5-9360-60f50fca71eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e62d8345-fe5a-49c5-9360-60f50fca71eb" (UID: "e62d8345-fe5a-49c5-9360-60f50fca71eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:57.394657 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.394633 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62d8345-fe5a-49c5-9360-60f50fca71eb-kube-api-access-qtwpg" (OuterVolumeSpecName: "kube-api-access-qtwpg") pod "e62d8345-fe5a-49c5-9360-60f50fca71eb" (UID: "e62d8345-fe5a-49c5-9360-60f50fca71eb"). InnerVolumeSpecName "kube-api-access-qtwpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:57.451459 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.451417 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d4a9503-62f8-43d5-b238-a6c536c8846a" path="/var/lib/kubelet/pods/5d4a9503-62f8-43d5-b238-a6c536c8846a/volumes" Sep 17 18:30:57.492393 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.492363 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da-plugins-config-volume\") pod \"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da\" (UID: \"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da\") " Sep 17 18:30:57.492515 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.492481 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z99kx\" (UniqueName: \"kubernetes.io/projected/7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da-kube-api-access-z99kx\") pod \"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da\" (UID: \"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da\") " Sep 17 18:30:57.492744 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.492729 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtwpg\" (UniqueName: \"kubernetes.io/projected/e62d8345-fe5a-49c5-9360-60f50fca71eb-kube-api-access-qtwpg\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:57.492805 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.492738 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da" (UID: "7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:57.492805 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.492749 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62d8345-fe5a-49c5-9360-60f50fca71eb-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:57.494601 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.494583 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da-kube-api-access-z99kx" (OuterVolumeSpecName: "kube-api-access-z99kx") pod "7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da" (UID: "7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da"). InnerVolumeSpecName "kube-api-access-z99kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:57.593880 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.593815 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z99kx\" (UniqueName: \"kubernetes.io/projected/7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da-kube-api-access-z99kx\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:57.593880 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.593847 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:30:57.728097 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.728058 2564 generic.go:356] "Generic (PLEG): container finished" podID="e62d8345-fe5a-49c5-9360-60f50fca71eb" containerID="85bec18417df1721b68606dd27d297dc0c752594c8a1eee50657afaf9f90b283" exitCode=0 Sep 17 18:30:57.728226 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.728113 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" Sep 17 18:30:57.728226 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.728145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" event={"ID":"e62d8345-fe5a-49c5-9360-60f50fca71eb","Type":"ContainerDied","Data":"85bec18417df1721b68606dd27d297dc0c752594c8a1eee50657afaf9f90b283"} Sep 17 18:30:57.728398 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.728288 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv" event={"ID":"e62d8345-fe5a-49c5-9360-60f50fca71eb","Type":"ContainerDied","Data":"ca47eba6f2a0b416e18a6712a3dad2d9f95fc9d6d3b59ed2eceb10d8f5ee5e9e"} Sep 17 18:30:57.729317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.728313 2564 scope.go:117] "RemoveContainer" containerID="85bec18417df1721b68606dd27d297dc0c752594c8a1eee50657afaf9f90b283" Sep 17 18:30:57.729921 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.729897 2564 generic.go:356] "Generic (PLEG): container finished" podID="7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da" containerID="dcb5097189c8306b0e8c99abc577850d532879f7e7b7d23d51b52b101f0dd1a8" exitCode=0 Sep 17 18:30:57.730039 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.729924 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" event={"ID":"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da","Type":"ContainerDied","Data":"dcb5097189c8306b0e8c99abc577850d532879f7e7b7d23d51b52b101f0dd1a8"} Sep 17 18:30:57.730039 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.729945 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" Sep 17 18:30:57.730039 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.729991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87" event={"ID":"7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da","Type":"ContainerDied","Data":"09f836ce7f53ee50e10d5fe6f2266ae1144527afec93812582cca3f79faa2667"} Sep 17 18:30:57.739578 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.739562 2564 scope.go:117] "RemoveContainer" containerID="85bec18417df1721b68606dd27d297dc0c752594c8a1eee50657afaf9f90b283" Sep 17 18:30:57.739846 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:57.739829 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85bec18417df1721b68606dd27d297dc0c752594c8a1eee50657afaf9f90b283\": container with ID starting with 85bec18417df1721b68606dd27d297dc0c752594c8a1eee50657afaf9f90b283 not found: ID does not exist" containerID="85bec18417df1721b68606dd27d297dc0c752594c8a1eee50657afaf9f90b283" Sep 17 18:30:57.739902 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.739852 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85bec18417df1721b68606dd27d297dc0c752594c8a1eee50657afaf9f90b283"} err="failed to get container status \"85bec18417df1721b68606dd27d297dc0c752594c8a1eee50657afaf9f90b283\": rpc error: code = NotFound desc = could not find container \"85bec18417df1721b68606dd27d297dc0c752594c8a1eee50657afaf9f90b283\": container with ID starting with 85bec18417df1721b68606dd27d297dc0c752594c8a1eee50657afaf9f90b283 not found: ID does not exist" Sep 17 18:30:57.739902 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.739867 2564 scope.go:117] "RemoveContainer" containerID="dcb5097189c8306b0e8c99abc577850d532879f7e7b7d23d51b52b101f0dd1a8" Sep 17 18:30:57.748103 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.748074 2564 scope.go:117] "RemoveContainer" containerID="dcb5097189c8306b0e8c99abc577850d532879f7e7b7d23d51b52b101f0dd1a8" Sep 17 18:30:57.748492 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:30:57.748438 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb5097189c8306b0e8c99abc577850d532879f7e7b7d23d51b52b101f0dd1a8\": container with ID starting with dcb5097189c8306b0e8c99abc577850d532879f7e7b7d23d51b52b101f0dd1a8 not found: ID does not exist" containerID="dcb5097189c8306b0e8c99abc577850d532879f7e7b7d23d51b52b101f0dd1a8" Sep 17 18:30:57.748492 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.748468 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb5097189c8306b0e8c99abc577850d532879f7e7b7d23d51b52b101f0dd1a8"} err="failed to get container status \"dcb5097189c8306b0e8c99abc577850d532879f7e7b7d23d51b52b101f0dd1a8\": rpc error: code = NotFound desc = could not find container \"dcb5097189c8306b0e8c99abc577850d532879f7e7b7d23d51b52b101f0dd1a8\": container with ID starting with dcb5097189c8306b0e8c99abc577850d532879f7e7b7d23d51b52b101f0dd1a8 not found: ID does not exist" Sep 17 18:30:57.751112 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.751093 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv"] Sep 17 18:30:57.757385 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.757365 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-03270e-5ff5c9d4bf-ccsdv"] Sep 17 18:30:57.766381 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.766362 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87"] Sep 17 18:30:57.770296 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:57.770277 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-03270e-76dcc585c4-lkg87"] Sep 17 18:30:59.450695 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:59.450665 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da" path="/var/lib/kubelet/pods/7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da/volumes" Sep 17 18:30:59.451054 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:30:59.451017 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e62d8345-fe5a-49c5-9360-60f50fca71eb" path="/var/lib/kubelet/pods/e62d8345-fe5a-49c5-9360-60f50fca71eb/volumes" Sep 17 18:34:41.452928 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.452899 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s"] Sep 17 18:34:41.453317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453260 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerName="istio-proxy" Sep 17 18:34:41.453317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453271 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerName="istio-proxy" Sep 17 18:34:41.453317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453280 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerName="istio-proxy" Sep 17 18:34:41.453317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453286 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="92979558-f5c7-46c0-9eb2-3cdaf8edb19d" containerName="istio-proxy" Sep 17 18:34:41.453317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453296 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerName="istio-proxy" Sep 17 18:34:41.453317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453301 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerName="istio-proxy" Sep 17 18:34:41.453317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453311 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d4a9503-62f8-43d5-b238-a6c536c8846a" containerName="revoke-keys" Sep 17 18:34:41.453317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453316 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d4a9503-62f8-43d5-b238-a6c536c8846a" containerName="revoke-keys" Sep 17 18:34:41.453317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453322 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453327 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453337 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e62d8345-fe5a-49c5-9360-60f50fca71eb" containerName="payload-processing" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453345 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e62d8345-fe5a-49c5-9360-60f50fca71eb" containerName="payload-processing" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453357 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerName="istio-proxy" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453366 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerName="istio-proxy" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453374 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da" containerName="payload-pre-processing" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453379 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da" containerName="payload-pre-processing" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453435 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e21c0cc6-1c7b-4d20-8fa1-2f01224de923" containerName="istio-proxy" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453448 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d4a9503-62f8-43d5-b238-a6c536c8846a" containerName="revoke-keys" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453454 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerName="istio-proxy" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453461 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c637b203-8705-4b9b-b5b5-9fa6728be0b8" containerName="istio-proxy" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453468 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e62d8345-fe5a-49c5-9360-60f50fca71eb" containerName="payload-processing" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453474 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d6d4d5e-9cab-4be7-b62e-6bb7b873d0da" containerName="payload-pre-processing" Sep 17 18:34:41.453899 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.453480 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ef44e5-6376-4ab4-960f-d6fa1104e301" containerName="istio-proxy" Sep 17 18:34:41.456596 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.456575 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.459147 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.459129 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f72155-openshift-default-dockercfg-fz8fj\"" Sep 17 18:34:41.468462 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.468438 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s"] Sep 17 18:34:41.547233 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.547205 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-workload-socket\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.547400 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.547238 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-credential-socket\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.547400 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.547295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzspb\" (UniqueName: \"kubernetes.io/projected/2a084ce3-4df0-474a-8bd2-d99e26fe8109-kube-api-access-mzspb\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.547400 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.547342 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-envoy\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.547400 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.547386 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-data\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.547608 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.547402 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-token\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.547608 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.547484 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istiod-ca-cert\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.547608 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.547513 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-workload-certs\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.547608 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.547602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-podinfo\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.648171 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.648141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-workload-socket\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.648171 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.648175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-credential-socket\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.648382 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.648197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzspb\" (UniqueName: \"kubernetes.io/projected/2a084ce3-4df0-474a-8bd2-d99e26fe8109-kube-api-access-mzspb\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.648382 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.648233 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-envoy\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.648382 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.648305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-data\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.648382 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.648330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-token\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.648382 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.648380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istiod-ca-cert\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.648699 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.648422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-workload-certs\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.648699 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.648480 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-podinfo\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.648699 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.648646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-credential-socket\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.648814 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.648750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-data\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.649074 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.649038 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-workload-certs\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.649172 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.649143 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istiod-ca-cert\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.649172 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.649146 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-workload-socket\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.651040 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.651007 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-envoy\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.651135 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.651094 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-podinfo\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.657244 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.657220 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-token\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.657468 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.657451 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzspb\" (UniqueName: \"kubernetes.io/projected/2a084ce3-4df0-474a-8bd2-d99e26fe8109-kube-api-access-mzspb\") pod \"e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:41.770561 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:41.770481 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:42.058684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:42.058654 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s"] Sep 17 18:34:42.059849 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:34:42.059822 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a084ce3_4df0_474a_8bd2_d99e26fe8109.slice/crio-735b040c91076cd63c52311aaac2e92cf1d21efeb9c5d2776fc83048c424a0bd WatchSource:0}: Error finding container 735b040c91076cd63c52311aaac2e92cf1d21efeb9c5d2776fc83048c424a0bd: Status 404 returned error can't find the container with id 735b040c91076cd63c52311aaac2e92cf1d21efeb9c5d2776fc83048c424a0bd Sep 17 18:34:42.062201 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:42.062154 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:34:42.062273 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:42.062256 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:34:42.062323 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:42.062300 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:34:43.051413 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:43.051364 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" event={"ID":"2a084ce3-4df0-474a-8bd2-d99e26fe8109","Type":"ContainerStarted","Data":"9f2f8737db0bf1a8d2f3a1480a07fc5bb295ad5b3f7dd28f428b92e9a2a89b74"} Sep 17 18:34:43.053853 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:43.051437 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" event={"ID":"2a084ce3-4df0-474a-8bd2-d99e26fe8109","Type":"ContainerStarted","Data":"735b040c91076cd63c52311aaac2e92cf1d21efeb9c5d2776fc83048c424a0bd"} Sep 17 18:34:43.082110 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:43.082064 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" podStartSLOduration=2.082049958 podStartE2EDuration="2.082049958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:34:43.080490905 +0000 UTC m=+1908.090678042" watchObservedRunningTime="2026-09-17 18:34:43.082049958 +0000 UTC m=+1908.092237094" Sep 17 18:34:43.771007 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:43.770977 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:43.775959 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:43.775932 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:44.061086 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:44.061058 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:44.062111 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:44.062082 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:34:47.555435 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:47.555394 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7"] Sep 17 18:34:47.559148 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:47.559131 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" Sep 17 18:34:47.561562 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:47.561537 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f72155\"" Sep 17 18:34:47.569341 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:47.569317 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7"] Sep 17 18:34:47.603068 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:47.603044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a60bada-5417-408a-8e07-8da36aa92a46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7\" (UID: \"0a60bada-5417-408a-8e07-8da36aa92a46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" Sep 17 18:34:47.603217 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:47.603078 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh5rt\" (UniqueName: \"kubernetes.io/projected/0a60bada-5417-408a-8e07-8da36aa92a46-kube-api-access-wh5rt\") pod \"payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7\" (UID: \"0a60bada-5417-408a-8e07-8da36aa92a46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" Sep 17 18:34:47.704294 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:47.704256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a60bada-5417-408a-8e07-8da36aa92a46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7\" (UID: \"0a60bada-5417-408a-8e07-8da36aa92a46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" Sep 17 18:34:47.704294 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:47.704299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh5rt\" (UniqueName: \"kubernetes.io/projected/0a60bada-5417-408a-8e07-8da36aa92a46-kube-api-access-wh5rt\") pod \"payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7\" (UID: \"0a60bada-5417-408a-8e07-8da36aa92a46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" Sep 17 18:34:47.705053 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:47.705031 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a60bada-5417-408a-8e07-8da36aa92a46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7\" (UID: \"0a60bada-5417-408a-8e07-8da36aa92a46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" Sep 17 18:34:47.717048 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:47.717014 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh5rt\" (UniqueName: \"kubernetes.io/projected/0a60bada-5417-408a-8e07-8da36aa92a46-kube-api-access-wh5rt\") pod \"payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7\" (UID: \"0a60bada-5417-408a-8e07-8da36aa92a46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" Sep 17 18:34:47.870695 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:47.870671 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" Sep 17 18:34:48.151797 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:48.151772 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7"] Sep 17 18:34:48.153726 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:34:48.153697 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a60bada_5417_408a_8e07_8da36aa92a46.slice/crio-8264f0ddff3c499171b3855fc95c1e0b2fdacb130330ce55a473e303b66c580c WatchSource:0}: Error finding container 8264f0ddff3c499171b3855fc95c1e0b2fdacb130330ce55a473e303b66c580c: Status 404 returned error can't find the container with id 8264f0ddff3c499171b3855fc95c1e0b2fdacb130330ce55a473e303b66c580c Sep 17 18:34:49.094808 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:49.094770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" event={"ID":"0a60bada-5417-408a-8e07-8da36aa92a46","Type":"ContainerStarted","Data":"bb37754a3d7e8ed67fafa5515c3dd35779b78116462408151fa2d7e52fb683df"} Sep 17 18:34:49.095177 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:49.094858 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" Sep 17 18:34:49.095177 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:49.094877 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" event={"ID":"0a60bada-5417-408a-8e07-8da36aa92a46","Type":"ContainerStarted","Data":"8264f0ddff3c499171b3855fc95c1e0b2fdacb130330ce55a473e303b66c580c"} Sep 17 18:34:49.114843 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:34:49.114801 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" podStartSLOduration=2.114781068 podStartE2EDuration="2.114781068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:34:49.112220624 +0000 UTC m=+1914.122407760" watchObservedRunningTime="2026-09-17 18:34:49.114781068 +0000 UTC m=+1914.124968240" Sep 17 18:35:00.102634 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:00.102602 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" Sep 17 18:35:00.103694 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:00.103660 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 17 18:35:00.103793 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:00.103731 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 18:35:02.182018 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:02.181955 2564 generic.go:356] "Generic (PLEG): container finished" podID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerID="9f2f8737db0bf1a8d2f3a1480a07fc5bb295ad5b3f7dd28f428b92e9a2a89b74" exitCode=0 Sep 17 18:35:02.182338 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:02.182034 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" event={"ID":"2a084ce3-4df0-474a-8bd2-d99e26fe8109","Type":"ContainerDied","Data":"9f2f8737db0bf1a8d2f3a1480a07fc5bb295ad5b3f7dd28f428b92e9a2a89b74"} Sep 17 18:35:02.182484 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:02.182467 2564 scope.go:117] "RemoveContainer" containerID="9f2f8737db0bf1a8d2f3a1480a07fc5bb295ad5b3f7dd28f428b92e9a2a89b74" Sep 17 18:35:02.184671 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:02.184008 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:35:02.184671 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:02.184104 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:35:02.184671 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:02.184149 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:35:03.191296 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:03.191263 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" event={"ID":"2a084ce3-4df0-474a-8bd2-d99e26fe8109","Type":"ContainerStarted","Data":"089daf10167bd6d6c88de860f20e088438aac93767a92e1dec7457f44c7e5878"} Sep 17 18:35:03.771408 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:03.771371 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:35:03.771621 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:03.771597 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 17 18:35:03.771699 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:03.771679 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 17 18:35:04.771923 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:04.771883 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 17 18:35:04.772391 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:04.771960 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 17 18:35:05.773514 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:05.773479 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 17 18:35:05.773999 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:05.773570 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 17 18:35:06.771204 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:06.771166 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 17 18:35:06.771375 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:06.771222 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 17 18:35:07.771654 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:07.771615 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 17 18:35:07.772053 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:07.771684 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 17 18:35:08.774516 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:08.774484 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:35:08.774998 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:08.774979 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:35:08.775748 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:08.775725 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:35:11.515452 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.515422 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf"] Sep 17 18:35:11.518084 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.518067 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.520865 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.520840 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0f4423-openshift-default-dockercfg-g7dd8\"" Sep 17 18:35:11.535743 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.535717 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf"] Sep 17 18:35:11.620183 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.620155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-workload-certs\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.620318 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.620200 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-token\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.620318 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.620221 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-credential-socket\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.620318 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.620283 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-podinfo\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.620318 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.620319 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-workload-socket\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.620586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.620333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-data\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.620586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.620349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-envoy\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.620586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.620372 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istiod-ca-cert\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.620586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.620418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g675\" (UniqueName: \"kubernetes.io/projected/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-kube-api-access-5g675\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.721406 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.721377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-workload-socket\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.721406 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.721407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-data\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.721671 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.721427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-envoy\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.721671 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.721456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istiod-ca-cert\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.721671 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.721490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g675\" (UniqueName: \"kubernetes.io/projected/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-kube-api-access-5g675\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.721671 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.721518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-workload-certs\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.721671 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.721590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-token\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.721671 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.721624 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-credential-socket\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.721987 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.721701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-podinfo\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.721987 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.721832 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-workload-socket\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.721987 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.721844 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-data\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.722191 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.722172 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-workload-certs\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.722273 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.722248 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-credential-socket\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.722333 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.722278 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istiod-ca-cert\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.723877 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.723856 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-envoy\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.724441 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.724416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-podinfo\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.735310 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.735284 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-token\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.741038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.741018 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g675\" (UniqueName: \"kubernetes.io/projected/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-kube-api-access-5g675\") pod \"e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:11.830396 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:11.829695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:12.110467 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:12.110435 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf"] Sep 17 18:35:12.111322 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:35:12.111297 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefcab250_f7ce_4d5f_9c1d_d62d6243ead6.slice/crio-a8e42b8e1ff38bcf528b4ac9f8bee692645d88c5a02c5b9b3c01e401045e6e6a WatchSource:0}: Error finding container a8e42b8e1ff38bcf528b4ac9f8bee692645d88c5a02c5b9b3c01e401045e6e6a: Status 404 returned error can't find the container with id a8e42b8e1ff38bcf528b4ac9f8bee692645d88c5a02c5b9b3c01e401045e6e6a Sep 17 18:35:12.113509 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:12.113471 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:35:12.113630 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:12.113565 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:35:12.113630 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:12.113602 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:35:12.248520 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:12.248483 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" event={"ID":"efcab250-f7ce-4d5f-9c1d-d62d6243ead6","Type":"ContainerStarted","Data":"a90ba830c314cd9f7ef4bb62bef486e763df0e875c08afae72c481eb499c2dfd"} Sep 17 18:35:12.248713 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:12.248561 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" event={"ID":"efcab250-f7ce-4d5f-9c1d-d62d6243ead6","Type":"ContainerStarted","Data":"a8e42b8e1ff38bcf528b4ac9f8bee692645d88c5a02c5b9b3c01e401045e6e6a"} Sep 17 18:35:12.293304 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:12.293246 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" podStartSLOduration=1.29322417 podStartE2EDuration="1.29322417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:12.289947193 +0000 UTC m=+1937.300134330" watchObservedRunningTime="2026-09-17 18:35:12.29322417 +0000 UTC m=+1937.303411306" Sep 17 18:35:12.830541 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:12.830506 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:13.836266 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:13.836232 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:14.260498 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:14.260418 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:14.261488 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:14.261468 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:17.809795 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:17.809762 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45"] Sep 17 18:35:17.812682 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:17.812666 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" Sep 17 18:35:17.815059 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:17.815036 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0f4423-dockercfg-sb2h4\"" Sep 17 18:35:17.815188 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:17.815082 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0f4423\"" Sep 17 18:35:17.854095 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:17.854061 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45"] Sep 17 18:35:17.988387 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:17.988350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e8195b-04eb-4c7e-8b2c-7c6472ae702d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45\" (UID: \"40e8195b-04eb-4c7e-8b2c-7c6472ae702d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" Sep 17 18:35:17.988612 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:17.988438 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnqxn\" (UniqueName: \"kubernetes.io/projected/40e8195b-04eb-4c7e-8b2c-7c6472ae702d-kube-api-access-rnqxn\") pod \"payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45\" (UID: \"40e8195b-04eb-4c7e-8b2c-7c6472ae702d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" Sep 17 18:35:18.089502 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:18.089466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e8195b-04eb-4c7e-8b2c-7c6472ae702d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45\" (UID: \"40e8195b-04eb-4c7e-8b2c-7c6472ae702d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" Sep 17 18:35:18.089679 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:18.089569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnqxn\" (UniqueName: \"kubernetes.io/projected/40e8195b-04eb-4c7e-8b2c-7c6472ae702d-kube-api-access-rnqxn\") pod \"payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45\" (UID: \"40e8195b-04eb-4c7e-8b2c-7c6472ae702d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" Sep 17 18:35:18.090176 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:18.090157 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e8195b-04eb-4c7e-8b2c-7c6472ae702d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45\" (UID: \"40e8195b-04eb-4c7e-8b2c-7c6472ae702d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" Sep 17 18:35:18.097424 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:18.097402 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnqxn\" (UniqueName: \"kubernetes.io/projected/40e8195b-04eb-4c7e-8b2c-7c6472ae702d-kube-api-access-rnqxn\") pod \"payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45\" (UID: \"40e8195b-04eb-4c7e-8b2c-7c6472ae702d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" Sep 17 18:35:18.123356 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:18.123332 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" Sep 17 18:35:18.398077 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:18.398051 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45"] Sep 17 18:35:18.399112 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:35:18.399088 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e8195b_04eb_4c7e_8b2c_7c6472ae702d.slice/crio-06c9379a3070af18bf278fff679966087cbb9a5d14e28598f5a592e1cb26300a WatchSource:0}: Error finding container 06c9379a3070af18bf278fff679966087cbb9a5d14e28598f5a592e1cb26300a: Status 404 returned error can't find the container with id 06c9379a3070af18bf278fff679966087cbb9a5d14e28598f5a592e1cb26300a Sep 17 18:35:19.295313 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:19.295279 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" event={"ID":"40e8195b-04eb-4c7e-8b2c-7c6472ae702d","Type":"ContainerStarted","Data":"2130546aba737dcc81d67368612bfa09902c4795db392026e8373d191c6c14c2"} Sep 17 18:35:19.295758 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:19.295330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" event={"ID":"40e8195b-04eb-4c7e-8b2c-7c6472ae702d","Type":"ContainerStarted","Data":"06c9379a3070af18bf278fff679966087cbb9a5d14e28598f5a592e1cb26300a"} Sep 17 18:35:19.295758 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:19.295351 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" Sep 17 18:35:19.316173 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:19.316125 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" podStartSLOduration=2.316109726 podStartE2EDuration="2.316109726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:19.314007225 +0000 UTC m=+1944.324194386" watchObservedRunningTime="2026-09-17 18:35:19.316109726 +0000 UTC m=+1944.326296862" Sep 17 18:35:30.261095 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:30.261063 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 17 18:35:30.261551 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:30.261122 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 18:35:30.303389 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:30.303365 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" Sep 17 18:35:32.384246 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:32.384215 2564 generic.go:356] "Generic (PLEG): container finished" podID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerID="a90ba830c314cd9f7ef4bb62bef486e763df0e875c08afae72c481eb499c2dfd" exitCode=0 Sep 17 18:35:32.384674 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:32.384284 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" event={"ID":"efcab250-f7ce-4d5f-9c1d-d62d6243ead6","Type":"ContainerDied","Data":"a90ba830c314cd9f7ef4bb62bef486e763df0e875c08afae72c481eb499c2dfd"} Sep 17 18:35:32.384727 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:32.384698 2564 scope.go:117] "RemoveContainer" containerID="a90ba830c314cd9f7ef4bb62bef486e763df0e875c08afae72c481eb499c2dfd" Sep 17 18:35:32.385756 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:32.385726 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:35:32.385861 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:32.385796 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:35:32.385861 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:32.385825 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 18:35:33.392347 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:33.392311 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" event={"ID":"efcab250-f7ce-4d5f-9c1d-d62d6243ead6","Type":"ContainerStarted","Data":"dbcba73d628aac3268fd34e686d751803b3b2a67832873c8f24467e5a9e7294f"} Sep 17 18:35:33.830585 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:33.830555 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:33.830774 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:33.830752 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 17 18:35:33.830834 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:33.830814 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 17 18:35:34.830893 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:34.830861 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 17 18:35:34.831303 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:34.830914 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 17 18:35:35.830677 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:35.830644 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 17 18:35:35.830889 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:35.830705 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 17 18:35:36.830628 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:36.830593 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 17 18:35:36.831097 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:36.830657 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 17 18:35:37.830743 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:37.830705 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 17 18:35:37.831228 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:37.830763 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 17 18:35:38.834153 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:38.834116 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:38.834575 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:38.834470 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:38.835328 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:38.835311 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:35:55.819128 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:35:55.819059 2564 scope.go:117] "RemoveContainer" containerID="453015822ddb127225ddd3f1904ea6a1a9d4a656ff02683ee31ebab73996605c" Sep 17 18:36:26.263691 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:26.263647 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2"] Sep 17 18:36:26.266280 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:26.266264 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" Sep 17 18:36:26.268920 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:26.268895 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-c37358\"" Sep 17 18:36:26.269045 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:26.268895 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-c37358-dockercfg-qth5b\"" Sep 17 18:36:26.278246 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:26.278223 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2"] Sep 17 18:36:26.291396 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:26.291375 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead0400b-5625-4a3e-a7b0-558f4288499f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2\" (UID: \"ead0400b-5625-4a3e-a7b0-558f4288499f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" Sep 17 18:36:26.291488 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:26.291411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l8r4\" (UniqueName: \"kubernetes.io/projected/ead0400b-5625-4a3e-a7b0-558f4288499f-kube-api-access-9l8r4\") pod \"payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2\" (UID: \"ead0400b-5625-4a3e-a7b0-558f4288499f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" Sep 17 18:36:26.392321 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:26.392283 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead0400b-5625-4a3e-a7b0-558f4288499f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2\" (UID: \"ead0400b-5625-4a3e-a7b0-558f4288499f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" Sep 17 18:36:26.392485 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:26.392333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l8r4\" (UniqueName: \"kubernetes.io/projected/ead0400b-5625-4a3e-a7b0-558f4288499f-kube-api-access-9l8r4\") pod \"payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2\" (UID: \"ead0400b-5625-4a3e-a7b0-558f4288499f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" Sep 17 18:36:26.393023 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:26.393004 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead0400b-5625-4a3e-a7b0-558f4288499f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2\" (UID: \"ead0400b-5625-4a3e-a7b0-558f4288499f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" Sep 17 18:36:26.402479 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:26.402454 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l8r4\" (UniqueName: \"kubernetes.io/projected/ead0400b-5625-4a3e-a7b0-558f4288499f-kube-api-access-9l8r4\") pod \"payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2\" (UID: \"ead0400b-5625-4a3e-a7b0-558f4288499f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" Sep 17 18:36:26.577440 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:26.577418 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" Sep 17 18:36:26.855655 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:26.855622 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2"] Sep 17 18:36:26.856130 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:36:26.856104 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead0400b_5625_4a3e_a7b0_558f4288499f.slice/crio-9dc27f289eacbf5f0babbf19e8a8cda004c949af62f249da4f868f66c2424195 WatchSource:0}: Error finding container 9dc27f289eacbf5f0babbf19e8a8cda004c949af62f249da4f868f66c2424195: Status 404 returned error can't find the container with id 9dc27f289eacbf5f0babbf19e8a8cda004c949af62f249da4f868f66c2424195 Sep 17 18:36:27.707056 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:27.707023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" event={"ID":"ead0400b-5625-4a3e-a7b0-558f4288499f","Type":"ContainerStarted","Data":"ff840a064dd40fdc3c6bb4830607105cb10df2c966ec67b027bfbeec164ba156"} Sep 17 18:36:27.707443 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:27.707075 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" event={"ID":"ead0400b-5625-4a3e-a7b0-558f4288499f","Type":"ContainerStarted","Data":"9dc27f289eacbf5f0babbf19e8a8cda004c949af62f249da4f868f66c2424195"} Sep 17 18:36:27.707443 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:27.707221 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" Sep 17 18:36:27.723682 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:27.723623 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" podStartSLOduration=1.723610297 podStartE2EDuration="1.723610297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:27.722183007 +0000 UTC m=+2012.732370143" watchObservedRunningTime="2026-09-17 18:36:27.723610297 +0000 UTC m=+2012.733797463" Sep 17 18:36:38.715061 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:38.715026 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" Sep 17 18:36:52.857540 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:52.857495 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w"] Sep 17 18:36:52.862569 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:52.862552 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w" Sep 17 18:36:52.864766 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:52.864743 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 18:36:52.864870 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:52.864753 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\"" Sep 17 18:36:52.869945 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:52.869911 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w"] Sep 17 18:36:53.021216 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:53.021188 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79660b27-2d6b-482b-a1e1-55fe9fba1cfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w\" (UID: \"79660b27-2d6b-482b-a1e1-55fe9fba1cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w" Sep 17 18:36:53.122482 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:53.122421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79660b27-2d6b-482b-a1e1-55fe9fba1cfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w\" (UID: \"79660b27-2d6b-482b-a1e1-55fe9fba1cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w" Sep 17 18:36:53.122992 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:53.122976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79660b27-2d6b-482b-a1e1-55fe9fba1cfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w\" (UID: \"79660b27-2d6b-482b-a1e1-55fe9fba1cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w" Sep 17 18:36:53.174144 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:53.174123 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w" Sep 17 18:36:53.449251 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:36:53.449223 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79660b27_2d6b_482b_a1e1_55fe9fba1cfa.slice/crio-a26a951d9df8a30aed400b58b73d1a7669e8cfaa50b2fdff3549f93a220b8d11 WatchSource:0}: Error finding container a26a951d9df8a30aed400b58b73d1a7669e8cfaa50b2fdff3549f93a220b8d11: Status 404 returned error can't find the container with id a26a951d9df8a30aed400b58b73d1a7669e8cfaa50b2fdff3549f93a220b8d11 Sep 17 18:36:53.453215 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:53.453190 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w"] Sep 17 18:36:53.865332 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:53.865288 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w" event={"ID":"79660b27-2d6b-482b-a1e1-55fe9fba1cfa","Type":"ContainerStarted","Data":"56d39e8352a4e5dd1ded22aef7d0886999cadf2e8f6c81da092cec65006611a7"} Sep 17 18:36:53.865742 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:53.865349 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w" event={"ID":"79660b27-2d6b-482b-a1e1-55fe9fba1cfa","Type":"ContainerStarted","Data":"a26a951d9df8a30aed400b58b73d1a7669e8cfaa50b2fdff3549f93a220b8d11"} Sep 17 18:36:53.883950 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:53.883904 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w" podStartSLOduration=1.883889059 podStartE2EDuration="1.883889059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:53.881131904 +0000 UTC m=+2038.891319041" watchObservedRunningTime="2026-09-17 18:36:53.883889059 +0000 UTC m=+2038.894076195" Sep 17 18:36:54.873449 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:54.873407 2564 generic.go:356] "Generic (PLEG): container finished" podID="79660b27-2d6b-482b-a1e1-55fe9fba1cfa" containerID="56d39e8352a4e5dd1ded22aef7d0886999cadf2e8f6c81da092cec65006611a7" exitCode=0 Sep 17 18:36:54.873825 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:54.873499 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w" event={"ID":"79660b27-2d6b-482b-a1e1-55fe9fba1cfa","Type":"ContainerDied","Data":"56d39e8352a4e5dd1ded22aef7d0886999cadf2e8f6c81da092cec65006611a7"} Sep 17 18:36:55.842632 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:55.842603 2564 scope.go:117] "RemoveContainer" containerID="75009309fac3badf1fc686fc542e0ab73680ed82b97f410a0eff14cbadfe669b" Sep 17 18:36:55.851881 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:55.851861 2564 scope.go:117] "RemoveContainer" containerID="86277945eaa5cc539db670b9e527883ce501d96efca4911672900d53bddf5694" Sep 17 18:36:55.860029 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:55.860009 2564 scope.go:117] "RemoveContainer" containerID="adb62eb4763c3411e02b97e651bd5d09e3ff499ec2b206783fa67faf46beed6a" Sep 17 18:36:56.037474 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:56.037451 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w" Sep 17 18:36:56.149892 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:56.149827 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79660b27-2d6b-482b-a1e1-55fe9fba1cfa-maas-api-service-ca\") pod \"79660b27-2d6b-482b-a1e1-55fe9fba1cfa\" (UID: \"79660b27-2d6b-482b-a1e1-55fe9fba1cfa\") " Sep 17 18:36:56.150200 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:56.150182 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79660b27-2d6b-482b-a1e1-55fe9fba1cfa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79660b27-2d6b-482b-a1e1-55fe9fba1cfa" (UID: "79660b27-2d6b-482b-a1e1-55fe9fba1cfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:36:56.250677 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:56.250653 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79660b27-2d6b-482b-a1e1-55fe9fba1cfa-maas-api-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:36:56.895377 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:56.895347 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w" Sep 17 18:36:56.895597 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:56.895382 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w" event={"ID":"79660b27-2d6b-482b-a1e1-55fe9fba1cfa","Type":"ContainerDied","Data":"a26a951d9df8a30aed400b58b73d1a7669e8cfaa50b2fdff3549f93a220b8d11"} Sep 17 18:36:56.895597 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:36:56.895443 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a26a951d9df8a30aed400b58b73d1a7669e8cfaa50b2fdff3549f93a220b8d11" Sep 17 18:37:03.993628 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:03.993586 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2"] Sep 17 18:37:03.994102 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:03.993804 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" podUID="ead0400b-5625-4a3e-a7b0-558f4288499f" containerName="payload-processing" containerID="cri-o://ff840a064dd40fdc3c6bb4830607105cb10df2c966ec67b027bfbeec164ba156" gracePeriod=30 Sep 17 18:37:04.251840 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.251780 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" Sep 17 18:37:04.305208 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.305181 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w"] Sep 17 18:37:04.309492 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.309472 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c37358-msx8w"] Sep 17 18:37:04.424240 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.424209 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead0400b-5625-4a3e-a7b0-558f4288499f-plugins-config-volume\") pod \"ead0400b-5625-4a3e-a7b0-558f4288499f\" (UID: \"ead0400b-5625-4a3e-a7b0-558f4288499f\") " Sep 17 18:37:04.424421 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.424262 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l8r4\" (UniqueName: \"kubernetes.io/projected/ead0400b-5625-4a3e-a7b0-558f4288499f-kube-api-access-9l8r4\") pod \"ead0400b-5625-4a3e-a7b0-558f4288499f\" (UID: \"ead0400b-5625-4a3e-a7b0-558f4288499f\") " Sep 17 18:37:04.424692 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.424665 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead0400b-5625-4a3e-a7b0-558f4288499f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ead0400b-5625-4a3e-a7b0-558f4288499f" (UID: "ead0400b-5625-4a3e-a7b0-558f4288499f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:37:04.426637 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.426616 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead0400b-5625-4a3e-a7b0-558f4288499f-kube-api-access-9l8r4" (OuterVolumeSpecName: "kube-api-access-9l8r4") pod "ead0400b-5625-4a3e-a7b0-558f4288499f" (UID: "ead0400b-5625-4a3e-a7b0-558f4288499f"). InnerVolumeSpecName "kube-api-access-9l8r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:37:04.525576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.525481 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead0400b-5625-4a3e-a7b0-558f4288499f-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:37:04.525576 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.525513 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l8r4\" (UniqueName: \"kubernetes.io/projected/ead0400b-5625-4a3e-a7b0-558f4288499f-kube-api-access-9l8r4\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:37:04.943374 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.943339 2564 generic.go:356] "Generic (PLEG): container finished" podID="ead0400b-5625-4a3e-a7b0-558f4288499f" containerID="ff840a064dd40fdc3c6bb4830607105cb10df2c966ec67b027bfbeec164ba156" exitCode=0 Sep 17 18:37:04.943559 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.943381 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" Sep 17 18:37:04.943559 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.943425 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" event={"ID":"ead0400b-5625-4a3e-a7b0-558f4288499f","Type":"ContainerDied","Data":"ff840a064dd40fdc3c6bb4830607105cb10df2c966ec67b027bfbeec164ba156"} Sep 17 18:37:04.943559 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.943474 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2" event={"ID":"ead0400b-5625-4a3e-a7b0-558f4288499f","Type":"ContainerDied","Data":"9dc27f289eacbf5f0babbf19e8a8cda004c949af62f249da4f868f66c2424195"} Sep 17 18:37:04.943559 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.943496 2564 scope.go:117] "RemoveContainer" containerID="ff840a064dd40fdc3c6bb4830607105cb10df2c966ec67b027bfbeec164ba156" Sep 17 18:37:04.953701 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.953687 2564 scope.go:117] "RemoveContainer" containerID="ff840a064dd40fdc3c6bb4830607105cb10df2c966ec67b027bfbeec164ba156" Sep 17 18:37:04.953972 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:37:04.953953 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff840a064dd40fdc3c6bb4830607105cb10df2c966ec67b027bfbeec164ba156\": container with ID starting with ff840a064dd40fdc3c6bb4830607105cb10df2c966ec67b027bfbeec164ba156 not found: ID does not exist" containerID="ff840a064dd40fdc3c6bb4830607105cb10df2c966ec67b027bfbeec164ba156" Sep 17 18:37:04.954021 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.953980 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff840a064dd40fdc3c6bb4830607105cb10df2c966ec67b027bfbeec164ba156"} err="failed to get container status \"ff840a064dd40fdc3c6bb4830607105cb10df2c966ec67b027bfbeec164ba156\": rpc error: code = NotFound desc = could not find container \"ff840a064dd40fdc3c6bb4830607105cb10df2c966ec67b027bfbeec164ba156\": container with ID starting with ff840a064dd40fdc3c6bb4830607105cb10df2c966ec67b027bfbeec164ba156 not found: ID does not exist" Sep 17 18:37:04.971413 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.971386 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2"] Sep 17 18:37:04.975192 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:04.975172 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c37358-79fdf68f97-5cph2"] Sep 17 18:37:05.451380 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:05.451353 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79660b27-2d6b-482b-a1e1-55fe9fba1cfa" path="/var/lib/kubelet/pods/79660b27-2d6b-482b-a1e1-55fe9fba1cfa/volumes" Sep 17 18:37:05.451765 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:05.451748 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ead0400b-5625-4a3e-a7b0-558f4288499f" path="/var/lib/kubelet/pods/ead0400b-5625-4a3e-a7b0-558f4288499f/volumes" Sep 17 18:37:14.708045 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.708004 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c"] Sep 17 18:37:14.708496 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.708413 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79660b27-2d6b-482b-a1e1-55fe9fba1cfa" containerName="revoke-keys" Sep 17 18:37:14.708496 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.708425 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="79660b27-2d6b-482b-a1e1-55fe9fba1cfa" containerName="revoke-keys" Sep 17 18:37:14.708496 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.708434 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead0400b-5625-4a3e-a7b0-558f4288499f" containerName="payload-processing" Sep 17 18:37:14.708496 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.708440 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead0400b-5625-4a3e-a7b0-558f4288499f" containerName="payload-processing" Sep 17 18:37:14.708666 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.708522 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ead0400b-5625-4a3e-a7b0-558f4288499f" containerName="payload-processing" Sep 17 18:37:14.708666 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.708549 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="79660b27-2d6b-482b-a1e1-55fe9fba1cfa" containerName="revoke-keys" Sep 17 18:37:14.712888 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.712868 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c" Sep 17 18:37:14.715430 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.715405 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 18:37:14.715552 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.715453 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\"" Sep 17 18:37:14.721148 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.721126 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c"] Sep 17 18:37:14.802411 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.802385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c\" (UID: \"5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c" Sep 17 18:37:14.903590 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.903523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c\" (UID: \"5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c" Sep 17 18:37:14.904092 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:14.904073 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c\" (UID: \"5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c" Sep 17 18:37:15.024846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:15.024796 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c" Sep 17 18:37:15.310515 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:15.310492 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c"] Sep 17 18:37:15.312351 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:37:15.312324 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efdb99d_9a7d_4e1b_8268_1b2ea183eeb6.slice/crio-a0e0a4e14d3fddb6eba12f980e7a5fdb579bfc7acdb4866c46eaf20c3ae7d2d5 WatchSource:0}: Error finding container a0e0a4e14d3fddb6eba12f980e7a5fdb579bfc7acdb4866c46eaf20c3ae7d2d5: Status 404 returned error can't find the container with id a0e0a4e14d3fddb6eba12f980e7a5fdb579bfc7acdb4866c46eaf20c3ae7d2d5 Sep 17 18:37:16.009343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:16.009312 2564 generic.go:356] "Generic (PLEG): container finished" podID="5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6" containerID="2996f902c72cb92208c23db24d0246393864bf66b928a3ddc330246f22bcbfe5" exitCode=0 Sep 17 18:37:16.009656 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:16.009392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c" event={"ID":"5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6","Type":"ContainerDied","Data":"2996f902c72cb92208c23db24d0246393864bf66b928a3ddc330246f22bcbfe5"} Sep 17 18:37:16.009656 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:16.009445 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c" event={"ID":"5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6","Type":"ContainerStarted","Data":"a0e0a4e14d3fddb6eba12f980e7a5fdb579bfc7acdb4866c46eaf20c3ae7d2d5"} Sep 17 18:37:17.151865 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:17.151840 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c" Sep 17 18:37:17.225191 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:17.225164 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6-maas-api-service-ca\") pod \"5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6\" (UID: \"5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6\") " Sep 17 18:37:17.225501 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:17.225478 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6" (UID: "5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:37:17.326643 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:17.326621 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6-maas-api-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:37:18.022715 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:18.022689 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c" Sep 17 18:37:18.022881 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:18.022686 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c" event={"ID":"5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6","Type":"ContainerDied","Data":"a0e0a4e14d3fddb6eba12f980e7a5fdb579bfc7acdb4866c46eaf20c3ae7d2d5"} Sep 17 18:37:18.022881 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:18.022820 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0e0a4e14d3fddb6eba12f980e7a5fdb579bfc7acdb4866c46eaf20c3ae7d2d5" Sep 17 18:37:30.377369 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:30.377333 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7"] Sep 17 18:37:30.378134 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:30.378082 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" podUID="0a60bada-5417-408a-8e07-8da36aa92a46" containerName="payload-pre-processing" containerID="cri-o://bb37754a3d7e8ed67fafa5515c3dd35779b78116462408151fa2d7e52fb683df" gracePeriod=30 Sep 17 18:37:30.575897 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:30.575863 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c"] Sep 17 18:37:30.582115 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:30.582088 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f72155-bhc4c"] Sep 17 18:37:30.653814 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:30.653786 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" Sep 17 18:37:30.742987 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:30.742858 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh5rt\" (UniqueName: \"kubernetes.io/projected/0a60bada-5417-408a-8e07-8da36aa92a46-kube-api-access-wh5rt\") pod \"0a60bada-5417-408a-8e07-8da36aa92a46\" (UID: \"0a60bada-5417-408a-8e07-8da36aa92a46\") " Sep 17 18:37:30.742987 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:30.742963 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a60bada-5417-408a-8e07-8da36aa92a46-plugins-config-volume\") pod \"0a60bada-5417-408a-8e07-8da36aa92a46\" (UID: \"0a60bada-5417-408a-8e07-8da36aa92a46\") " Sep 17 18:37:30.744217 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:30.744049 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a60bada-5417-408a-8e07-8da36aa92a46-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a60bada-5417-408a-8e07-8da36aa92a46" (UID: "0a60bada-5417-408a-8e07-8da36aa92a46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:37:30.745889 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:30.745866 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a60bada-5417-408a-8e07-8da36aa92a46-kube-api-access-wh5rt" (OuterVolumeSpecName: "kube-api-access-wh5rt") pod "0a60bada-5417-408a-8e07-8da36aa92a46" (UID: "0a60bada-5417-408a-8e07-8da36aa92a46"). InnerVolumeSpecName "kube-api-access-wh5rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:37:30.844738 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:30.844706 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh5rt\" (UniqueName: \"kubernetes.io/projected/0a60bada-5417-408a-8e07-8da36aa92a46-kube-api-access-wh5rt\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:37:30.844738 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:30.844737 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a60bada-5417-408a-8e07-8da36aa92a46-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:37:31.103296 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:31.103260 2564 generic.go:356] "Generic (PLEG): container finished" podID="0a60bada-5417-408a-8e07-8da36aa92a46" containerID="bb37754a3d7e8ed67fafa5515c3dd35779b78116462408151fa2d7e52fb683df" exitCode=0 Sep 17 18:37:31.103460 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:31.103302 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" Sep 17 18:37:31.103460 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:31.103303 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" event={"ID":"0a60bada-5417-408a-8e07-8da36aa92a46","Type":"ContainerDied","Data":"bb37754a3d7e8ed67fafa5515c3dd35779b78116462408151fa2d7e52fb683df"} Sep 17 18:37:31.103460 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:31.103396 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7" event={"ID":"0a60bada-5417-408a-8e07-8da36aa92a46","Type":"ContainerDied","Data":"8264f0ddff3c499171b3855fc95c1e0b2fdacb130330ce55a473e303b66c580c"} Sep 17 18:37:31.103460 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:31.103424 2564 scope.go:117] "RemoveContainer" containerID="bb37754a3d7e8ed67fafa5515c3dd35779b78116462408151fa2d7e52fb683df" Sep 17 18:37:31.120306 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:31.120287 2564 scope.go:117] "RemoveContainer" containerID="bb37754a3d7e8ed67fafa5515c3dd35779b78116462408151fa2d7e52fb683df" Sep 17 18:37:31.120636 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:37:31.120613 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb37754a3d7e8ed67fafa5515c3dd35779b78116462408151fa2d7e52fb683df\": container with ID starting with bb37754a3d7e8ed67fafa5515c3dd35779b78116462408151fa2d7e52fb683df not found: ID does not exist" containerID="bb37754a3d7e8ed67fafa5515c3dd35779b78116462408151fa2d7e52fb683df" Sep 17 18:37:31.120717 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:31.120648 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb37754a3d7e8ed67fafa5515c3dd35779b78116462408151fa2d7e52fb683df"} err="failed to get container status \"bb37754a3d7e8ed67fafa5515c3dd35779b78116462408151fa2d7e52fb683df\": rpc error: code = NotFound desc = could not find container \"bb37754a3d7e8ed67fafa5515c3dd35779b78116462408151fa2d7e52fb683df\": container with ID starting with bb37754a3d7e8ed67fafa5515c3dd35779b78116462408151fa2d7e52fb683df not found: ID does not exist" Sep 17 18:37:31.135183 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:31.135159 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7"] Sep 17 18:37:31.141003 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:31.140982 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f72155-8c78f8c78-68rm7"] Sep 17 18:37:31.452005 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:31.451916 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a60bada-5417-408a-8e07-8da36aa92a46" path="/var/lib/kubelet/pods/0a60bada-5417-408a-8e07-8da36aa92a46/volumes" Sep 17 18:37:31.452459 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:37:31.452433 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6" path="/var/lib/kubelet/pods/5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6/volumes" Sep 17 18:38:26.621738 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:26.621701 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s"] Sep 17 18:38:26.622221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:26.621902 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerName="istio-proxy" containerID="cri-o://089daf10167bd6d6c88de860f20e088438aac93767a92e1dec7457f44c7e5878" gracePeriod=30 Sep 17 18:38:31.877674 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.877647 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:38:31.949317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949250 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-credential-socket\") pod \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " Sep 17 18:38:31.949317 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949315 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-workload-socket\") pod \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " Sep 17 18:38:31.949508 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949365 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzspb\" (UniqueName: \"kubernetes.io/projected/2a084ce3-4df0-474a-8bd2-d99e26fe8109-kube-api-access-mzspb\") pod \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " Sep 17 18:38:31.949508 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949422 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-data\") pod \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " Sep 17 18:38:31.949508 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949448 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-token\") pod \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " Sep 17 18:38:31.949508 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949487 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-podinfo\") pod \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " Sep 17 18:38:31.949776 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949517 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-workload-certs\") pod \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " Sep 17 18:38:31.949776 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949549 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a084ce3-4df0-474a-8bd2-d99e26fe8109" (UID: "2a084ce3-4df0-474a-8bd2-d99e26fe8109"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:38:31.949776 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949581 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istiod-ca-cert\") pod \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " Sep 17 18:38:31.949776 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949602 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a084ce3-4df0-474a-8bd2-d99e26fe8109" (UID: "2a084ce3-4df0-474a-8bd2-d99e26fe8109"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:38:31.949776 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949618 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-envoy\") pod \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\" (UID: \"2a084ce3-4df0-474a-8bd2-d99e26fe8109\") " Sep 17 18:38:31.949776 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949703 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a084ce3-4df0-474a-8bd2-d99e26fe8109" (UID: "2a084ce3-4df0-474a-8bd2-d99e26fe8109"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:38:31.950098 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949829 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a084ce3-4df0-474a-8bd2-d99e26fe8109" (UID: "2a084ce3-4df0-474a-8bd2-d99e26fe8109"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:38:31.950098 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.949990 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:31.950098 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.950009 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:31.950098 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.950022 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:31.950098 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.950036 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:31.950098 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.950013 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a084ce3-4df0-474a-8bd2-d99e26fe8109" (UID: "2a084ce3-4df0-474a-8bd2-d99e26fe8109"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:38:31.951872 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.951847 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a084ce3-4df0-474a-8bd2-d99e26fe8109" (UID: "2a084ce3-4df0-474a-8bd2-d99e26fe8109"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:38:31.952015 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.951884 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a084ce3-4df0-474a-8bd2-d99e26fe8109-kube-api-access-mzspb" (OuterVolumeSpecName: "kube-api-access-mzspb") pod "2a084ce3-4df0-474a-8bd2-d99e26fe8109" (UID: "2a084ce3-4df0-474a-8bd2-d99e26fe8109"). InnerVolumeSpecName "kube-api-access-mzspb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:38:31.952323 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.952297 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a084ce3-4df0-474a-8bd2-d99e26fe8109" (UID: "2a084ce3-4df0-474a-8bd2-d99e26fe8109"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:38:31.952413 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:31.952321 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a084ce3-4df0-474a-8bd2-d99e26fe8109" (UID: "2a084ce3-4df0-474a-8bd2-d99e26fe8109"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:38:32.050761 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.050735 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:32.050761 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.050758 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:32.050900 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.050766 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:32.050900 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.050775 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a084ce3-4df0-474a-8bd2-d99e26fe8109-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:32.050900 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.050782 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzspb\" (UniqueName: \"kubernetes.io/projected/2a084ce3-4df0-474a-8bd2-d99e26fe8109-kube-api-access-mzspb\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:32.483806 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.483776 2564 generic.go:356] "Generic (PLEG): container finished" podID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerID="089daf10167bd6d6c88de860f20e088438aac93767a92e1dec7457f44c7e5878" exitCode=0 Sep 17 18:38:32.483944 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.483826 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" Sep 17 18:38:32.483944 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.483846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" event={"ID":"2a084ce3-4df0-474a-8bd2-d99e26fe8109","Type":"ContainerDied","Data":"089daf10167bd6d6c88de860f20e088438aac93767a92e1dec7457f44c7e5878"} Sep 17 18:38:32.483944 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.483896 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s" event={"ID":"2a084ce3-4df0-474a-8bd2-d99e26fe8109","Type":"ContainerDied","Data":"735b040c91076cd63c52311aaac2e92cf1d21efeb9c5d2776fc83048c424a0bd"} Sep 17 18:38:32.483944 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.483926 2564 scope.go:117] "RemoveContainer" containerID="089daf10167bd6d6c88de860f20e088438aac93767a92e1dec7457f44c7e5878" Sep 17 18:38:32.493880 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.493865 2564 scope.go:117] "RemoveContainer" containerID="9f2f8737db0bf1a8d2f3a1480a07fc5bb295ad5b3f7dd28f428b92e9a2a89b74" Sep 17 18:38:32.502994 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.502972 2564 scope.go:117] "RemoveContainer" containerID="089daf10167bd6d6c88de860f20e088438aac93767a92e1dec7457f44c7e5878" Sep 17 18:38:32.503255 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:38:32.503233 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089daf10167bd6d6c88de860f20e088438aac93767a92e1dec7457f44c7e5878\": container with ID starting with 089daf10167bd6d6c88de860f20e088438aac93767a92e1dec7457f44c7e5878 not found: ID does not exist" containerID="089daf10167bd6d6c88de860f20e088438aac93767a92e1dec7457f44c7e5878" Sep 17 18:38:32.503324 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.503263 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089daf10167bd6d6c88de860f20e088438aac93767a92e1dec7457f44c7e5878"} err="failed to get container status \"089daf10167bd6d6c88de860f20e088438aac93767a92e1dec7457f44c7e5878\": rpc error: code = NotFound desc = could not find container \"089daf10167bd6d6c88de860f20e088438aac93767a92e1dec7457f44c7e5878\": container with ID starting with 089daf10167bd6d6c88de860f20e088438aac93767a92e1dec7457f44c7e5878 not found: ID does not exist" Sep 17 18:38:32.503324 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.503280 2564 scope.go:117] "RemoveContainer" containerID="9f2f8737db0bf1a8d2f3a1480a07fc5bb295ad5b3f7dd28f428b92e9a2a89b74" Sep 17 18:38:32.503583 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:38:32.503569 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2f8737db0bf1a8d2f3a1480a07fc5bb295ad5b3f7dd28f428b92e9a2a89b74\": container with ID starting with 9f2f8737db0bf1a8d2f3a1480a07fc5bb295ad5b3f7dd28f428b92e9a2a89b74 not found: ID does not exist" containerID="9f2f8737db0bf1a8d2f3a1480a07fc5bb295ad5b3f7dd28f428b92e9a2a89b74" Sep 17 18:38:32.503640 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.503589 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2f8737db0bf1a8d2f3a1480a07fc5bb295ad5b3f7dd28f428b92e9a2a89b74"} err="failed to get container status \"9f2f8737db0bf1a8d2f3a1480a07fc5bb295ad5b3f7dd28f428b92e9a2a89b74\": rpc error: code = NotFound desc = could not find container \"9f2f8737db0bf1a8d2f3a1480a07fc5bb295ad5b3f7dd28f428b92e9a2a89b74\": container with ID starting with 9f2f8737db0bf1a8d2f3a1480a07fc5bb295ad5b3f7dd28f428b92e9a2a89b74 not found: ID does not exist" Sep 17 18:38:32.514299 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.514278 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s"] Sep 17 18:38:32.517191 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:32.517169 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f72155-openshift-default-6fdd7dc55d-ftz5s"] Sep 17 18:38:33.451291 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:33.451247 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" path="/var/lib/kubelet/pods/2a084ce3-4df0-474a-8bd2-d99e26fe8109/volumes" Sep 17 18:38:38.038438 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.038401 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45"] Sep 17 18:38:38.039056 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.038806 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" podUID="40e8195b-04eb-4c7e-8b2c-7c6472ae702d" containerName="payload-pre-processing" containerID="cri-o://2130546aba737dcc81d67368612bfa09902c4795db392026e8373d191c6c14c2" gracePeriod=30 Sep 17 18:38:38.297227 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.297172 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" Sep 17 18:38:38.406218 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.406164 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnqxn\" (UniqueName: \"kubernetes.io/projected/40e8195b-04eb-4c7e-8b2c-7c6472ae702d-kube-api-access-rnqxn\") pod \"40e8195b-04eb-4c7e-8b2c-7c6472ae702d\" (UID: \"40e8195b-04eb-4c7e-8b2c-7c6472ae702d\") " Sep 17 18:38:38.406696 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.406670 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e8195b-04eb-4c7e-8b2c-7c6472ae702d-plugins-config-volume\") pod \"40e8195b-04eb-4c7e-8b2c-7c6472ae702d\" (UID: \"40e8195b-04eb-4c7e-8b2c-7c6472ae702d\") " Sep 17 18:38:38.407304 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.407264 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e8195b-04eb-4c7e-8b2c-7c6472ae702d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40e8195b-04eb-4c7e-8b2c-7c6472ae702d" (UID: "40e8195b-04eb-4c7e-8b2c-7c6472ae702d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:38:38.408884 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.408856 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e8195b-04eb-4c7e-8b2c-7c6472ae702d-kube-api-access-rnqxn" (OuterVolumeSpecName: "kube-api-access-rnqxn") pod "40e8195b-04eb-4c7e-8b2c-7c6472ae702d" (UID: "40e8195b-04eb-4c7e-8b2c-7c6472ae702d"). InnerVolumeSpecName "kube-api-access-rnqxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:38:38.508404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.508375 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnqxn\" (UniqueName: \"kubernetes.io/projected/40e8195b-04eb-4c7e-8b2c-7c6472ae702d-kube-api-access-rnqxn\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:38.508404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.508400 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e8195b-04eb-4c7e-8b2c-7c6472ae702d-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:38.525773 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.525739 2564 generic.go:356] "Generic (PLEG): container finished" podID="40e8195b-04eb-4c7e-8b2c-7c6472ae702d" containerID="2130546aba737dcc81d67368612bfa09902c4795db392026e8373d191c6c14c2" exitCode=0 Sep 17 18:38:38.525860 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.525781 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" Sep 17 18:38:38.525860 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.525827 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" event={"ID":"40e8195b-04eb-4c7e-8b2c-7c6472ae702d","Type":"ContainerDied","Data":"2130546aba737dcc81d67368612bfa09902c4795db392026e8373d191c6c14c2"} Sep 17 18:38:38.525946 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.525872 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45" event={"ID":"40e8195b-04eb-4c7e-8b2c-7c6472ae702d","Type":"ContainerDied","Data":"06c9379a3070af18bf278fff679966087cbb9a5d14e28598f5a592e1cb26300a"} Sep 17 18:38:38.525946 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.525894 2564 scope.go:117] "RemoveContainer" containerID="2130546aba737dcc81d67368612bfa09902c4795db392026e8373d191c6c14c2" Sep 17 18:38:38.536415 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.536398 2564 scope.go:117] "RemoveContainer" containerID="2130546aba737dcc81d67368612bfa09902c4795db392026e8373d191c6c14c2" Sep 17 18:38:38.536689 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:38:38.536672 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2130546aba737dcc81d67368612bfa09902c4795db392026e8373d191c6c14c2\": container with ID starting with 2130546aba737dcc81d67368612bfa09902c4795db392026e8373d191c6c14c2 not found: ID does not exist" containerID="2130546aba737dcc81d67368612bfa09902c4795db392026e8373d191c6c14c2" Sep 17 18:38:38.536761 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.536702 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2130546aba737dcc81d67368612bfa09902c4795db392026e8373d191c6c14c2"} err="failed to get container status \"2130546aba737dcc81d67368612bfa09902c4795db392026e8373d191c6c14c2\": rpc error: code = NotFound desc = could not find container \"2130546aba737dcc81d67368612bfa09902c4795db392026e8373d191c6c14c2\": container with ID starting with 2130546aba737dcc81d67368612bfa09902c4795db392026e8373d191c6c14c2 not found: ID does not exist" Sep 17 18:38:38.578777 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.578756 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45"] Sep 17 18:38:38.592868 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:38.592850 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f4423-6cbf66b598-47v45"] Sep 17 18:38:39.451166 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:39.451134 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40e8195b-04eb-4c7e-8b2c-7c6472ae702d" path="/var/lib/kubelet/pods/40e8195b-04eb-4c7e-8b2c-7c6472ae702d/volumes" Sep 17 18:38:45.340002 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:45.339960 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf"] Sep 17 18:38:45.340455 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:45.340220 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerName="istio-proxy" containerID="cri-o://dbcba73d628aac3268fd34e686d751803b3b2a67832873c8f24467e5a9e7294f" gracePeriod=30 Sep 17 18:38:50.588344 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.588321 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:38:50.596753 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.596729 2564 generic.go:356] "Generic (PLEG): container finished" podID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerID="dbcba73d628aac3268fd34e686d751803b3b2a67832873c8f24467e5a9e7294f" exitCode=0 Sep 17 18:38:50.596883 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.596776 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" Sep 17 18:38:50.596883 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.596817 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" event={"ID":"efcab250-f7ce-4d5f-9c1d-d62d6243ead6","Type":"ContainerDied","Data":"dbcba73d628aac3268fd34e686d751803b3b2a67832873c8f24467e5a9e7294f"} Sep 17 18:38:50.596883 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.596873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf" event={"ID":"efcab250-f7ce-4d5f-9c1d-d62d6243ead6","Type":"ContainerDied","Data":"a8e42b8e1ff38bcf528b4ac9f8bee692645d88c5a02c5b9b3c01e401045e6e6a"} Sep 17 18:38:50.597000 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.596897 2564 scope.go:117] "RemoveContainer" containerID="dbcba73d628aac3268fd34e686d751803b3b2a67832873c8f24467e5a9e7294f" Sep 17 18:38:50.606066 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.606048 2564 scope.go:117] "RemoveContainer" containerID="a90ba830c314cd9f7ef4bb62bef486e763df0e875c08afae72c481eb499c2dfd" Sep 17 18:38:50.619989 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.619963 2564 scope.go:117] "RemoveContainer" containerID="dbcba73d628aac3268fd34e686d751803b3b2a67832873c8f24467e5a9e7294f" Sep 17 18:38:50.620295 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:38:50.620274 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbcba73d628aac3268fd34e686d751803b3b2a67832873c8f24467e5a9e7294f\": container with ID starting with dbcba73d628aac3268fd34e686d751803b3b2a67832873c8f24467e5a9e7294f not found: ID does not exist" containerID="dbcba73d628aac3268fd34e686d751803b3b2a67832873c8f24467e5a9e7294f" Sep 17 18:38:50.620351 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.620302 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbcba73d628aac3268fd34e686d751803b3b2a67832873c8f24467e5a9e7294f"} err="failed to get container status \"dbcba73d628aac3268fd34e686d751803b3b2a67832873c8f24467e5a9e7294f\": rpc error: code = NotFound desc = could not find container \"dbcba73d628aac3268fd34e686d751803b3b2a67832873c8f24467e5a9e7294f\": container with ID starting with dbcba73d628aac3268fd34e686d751803b3b2a67832873c8f24467e5a9e7294f not found: ID does not exist" Sep 17 18:38:50.620351 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.620318 2564 scope.go:117] "RemoveContainer" containerID="a90ba830c314cd9f7ef4bb62bef486e763df0e875c08afae72c481eb499c2dfd" Sep 17 18:38:50.620588 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:38:50.620569 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90ba830c314cd9f7ef4bb62bef486e763df0e875c08afae72c481eb499c2dfd\": container with ID starting with a90ba830c314cd9f7ef4bb62bef486e763df0e875c08afae72c481eb499c2dfd not found: ID does not exist" containerID="a90ba830c314cd9f7ef4bb62bef486e763df0e875c08afae72c481eb499c2dfd" Sep 17 18:38:50.620659 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.620592 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90ba830c314cd9f7ef4bb62bef486e763df0e875c08afae72c481eb499c2dfd"} err="failed to get container status \"a90ba830c314cd9f7ef4bb62bef486e763df0e875c08afae72c481eb499c2dfd\": rpc error: code = NotFound desc = could not find container \"a90ba830c314cd9f7ef4bb62bef486e763df0e875c08afae72c481eb499c2dfd\": container with ID starting with a90ba830c314cd9f7ef4bb62bef486e763df0e875c08afae72c481eb499c2dfd not found: ID does not exist" Sep 17 18:38:50.707714 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.707694 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-workload-socket\") pod \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " Sep 17 18:38:50.707804 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.707721 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g675\" (UniqueName: \"kubernetes.io/projected/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-kube-api-access-5g675\") pod \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " Sep 17 18:38:50.707804 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.707751 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-workload-certs\") pod \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " Sep 17 18:38:50.707804 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.707788 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istiod-ca-cert\") pod \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " Sep 17 18:38:50.707958 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.707837 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-token\") pod \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " Sep 17 18:38:50.707958 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.707883 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-data\") pod \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " Sep 17 18:38:50.707958 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.707953 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-envoy\") pod \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " Sep 17 18:38:50.708117 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.707972 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "efcab250-f7ce-4d5f-9c1d-d62d6243ead6" (UID: "efcab250-f7ce-4d5f-9c1d-d62d6243ead6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:38:50.708117 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.707977 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-credential-socket\") pod \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " Sep 17 18:38:50.708117 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.708032 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "efcab250-f7ce-4d5f-9c1d-d62d6243ead6" (UID: "efcab250-f7ce-4d5f-9c1d-d62d6243ead6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:38:50.708117 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.708042 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-podinfo\") pod \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\" (UID: \"efcab250-f7ce-4d5f-9c1d-d62d6243ead6\") " Sep 17 18:38:50.708334 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.708267 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-data" (OuterVolumeSpecName: "istio-data") pod "efcab250-f7ce-4d5f-9c1d-d62d6243ead6" (UID: "efcab250-f7ce-4d5f-9c1d-d62d6243ead6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:38:50.708391 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.708343 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "efcab250-f7ce-4d5f-9c1d-d62d6243ead6" (UID: "efcab250-f7ce-4d5f-9c1d-d62d6243ead6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:38:50.708609 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.708578 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:50.708609 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.708607 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:50.708742 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.708620 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:50.708742 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.708632 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:50.708863 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.708718 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "efcab250-f7ce-4d5f-9c1d-d62d6243ead6" (UID: "efcab250-f7ce-4d5f-9c1d-d62d6243ead6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:38:50.710258 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.710236 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "efcab250-f7ce-4d5f-9c1d-d62d6243ead6" (UID: "efcab250-f7ce-4d5f-9c1d-d62d6243ead6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:38:50.710370 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.710353 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-kube-api-access-5g675" (OuterVolumeSpecName: "kube-api-access-5g675") pod "efcab250-f7ce-4d5f-9c1d-d62d6243ead6" (UID: "efcab250-f7ce-4d5f-9c1d-d62d6243ead6"). InnerVolumeSpecName "kube-api-access-5g675". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:38:50.710432 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.710410 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-token" (OuterVolumeSpecName: "istio-token") pod "efcab250-f7ce-4d5f-9c1d-d62d6243ead6" (UID: "efcab250-f7ce-4d5f-9c1d-d62d6243ead6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:38:50.710613 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.710599 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "efcab250-f7ce-4d5f-9c1d-d62d6243ead6" (UID: "efcab250-f7ce-4d5f-9c1d-d62d6243ead6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:38:50.809017 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.808995 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g675\" (UniqueName: \"kubernetes.io/projected/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-kube-api-access-5g675\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:50.809017 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.809017 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:50.809132 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.809026 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:50.809132 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.809034 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:50.809132 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.809042 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efcab250-f7ce-4d5f-9c1d-d62d6243ead6-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:50.924927 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.924899 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf"] Sep 17 18:38:50.927322 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:50.927305 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f4423-openshift-default-5cd99f98b6-fdbsf"] Sep 17 18:38:51.450752 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:51.450727 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" path="/var/lib/kubelet/pods/efcab250-f7ce-4d5f-9c1d-d62d6243ead6/volumes" Sep 17 18:38:56.577850 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:56.577814 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2"] Sep 17 18:38:56.583157 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:56.583130 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-95a92b-29827830-7j9c2"] Sep 17 18:38:56.606846 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:56.606814 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc"] Sep 17 18:38:56.607065 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:56.607036 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" podUID="9c8d7e85-838b-4210-b470-2d1972e63bf6" containerName="payload-processing" containerID="cri-o://34cde97c73c3392ce9dc6b82b9caf3d6fbe57c30ed93b35a399c312f08b95f16" gracePeriod=30 Sep 17 18:38:56.629888 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:56.629838 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm"] Sep 17 18:38:56.630139 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:56.630095 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" podUID="16834e97-f84f-48af-b796-607dcef66fb4" containerName="payload-pre-processing" containerID="cri-o://f06fcfdfba05af0f8471e73e03ead05a435653d9090c1b8003995493cf11201b" gracePeriod=30 Sep 17 18:38:56.909819 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:56.909796 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" Sep 17 18:38:56.912751 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:56.912734 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" Sep 17 18:38:57.062023 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.061991 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2phl8\" (UniqueName: \"kubernetes.io/projected/16834e97-f84f-48af-b796-607dcef66fb4-kube-api-access-2phl8\") pod \"16834e97-f84f-48af-b796-607dcef66fb4\" (UID: \"16834e97-f84f-48af-b796-607dcef66fb4\") " Sep 17 18:38:57.062192 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.062049 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8d7e85-838b-4210-b470-2d1972e63bf6-plugins-config-volume\") pod \"9c8d7e85-838b-4210-b470-2d1972e63bf6\" (UID: \"9c8d7e85-838b-4210-b470-2d1972e63bf6\") " Sep 17 18:38:57.062192 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.062116 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8lvq\" (UniqueName: \"kubernetes.io/projected/9c8d7e85-838b-4210-b470-2d1972e63bf6-kube-api-access-f8lvq\") pod \"9c8d7e85-838b-4210-b470-2d1972e63bf6\" (UID: \"9c8d7e85-838b-4210-b470-2d1972e63bf6\") " Sep 17 18:38:57.062192 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.062135 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16834e97-f84f-48af-b796-607dcef66fb4-plugins-config-volume\") pod \"16834e97-f84f-48af-b796-607dcef66fb4\" (UID: \"16834e97-f84f-48af-b796-607dcef66fb4\") " Sep 17 18:38:57.062475 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.062451 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8d7e85-838b-4210-b470-2d1972e63bf6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c8d7e85-838b-4210-b470-2d1972e63bf6" (UID: "9c8d7e85-838b-4210-b470-2d1972e63bf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:38:57.062611 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.062491 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16834e97-f84f-48af-b796-607dcef66fb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16834e97-f84f-48af-b796-607dcef66fb4" (UID: "16834e97-f84f-48af-b796-607dcef66fb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:38:57.064361 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.064344 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16834e97-f84f-48af-b796-607dcef66fb4-kube-api-access-2phl8" (OuterVolumeSpecName: "kube-api-access-2phl8") pod "16834e97-f84f-48af-b796-607dcef66fb4" (UID: "16834e97-f84f-48af-b796-607dcef66fb4"). InnerVolumeSpecName "kube-api-access-2phl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:38:57.064489 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.064471 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8d7e85-838b-4210-b470-2d1972e63bf6-kube-api-access-f8lvq" (OuterVolumeSpecName: "kube-api-access-f8lvq") pod "9c8d7e85-838b-4210-b470-2d1972e63bf6" (UID: "9c8d7e85-838b-4210-b470-2d1972e63bf6"). InnerVolumeSpecName "kube-api-access-f8lvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:38:57.163214 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.163147 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8d7e85-838b-4210-b470-2d1972e63bf6-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:57.163214 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.163179 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8lvq\" (UniqueName: \"kubernetes.io/projected/9c8d7e85-838b-4210-b470-2d1972e63bf6-kube-api-access-f8lvq\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:57.163214 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.163189 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16834e97-f84f-48af-b796-607dcef66fb4-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:57.163214 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.163198 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2phl8\" (UniqueName: \"kubernetes.io/projected/16834e97-f84f-48af-b796-607dcef66fb4-kube-api-access-2phl8\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:38:57.450751 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.450683 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12795c6d-6d9c-4ce8-a6be-decbb9f9af6a" path="/var/lib/kubelet/pods/12795c6d-6d9c-4ce8-a6be-decbb9f9af6a/volumes" Sep 17 18:38:57.649171 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.649138 2564 generic.go:356] "Generic (PLEG): container finished" podID="9c8d7e85-838b-4210-b470-2d1972e63bf6" containerID="34cde97c73c3392ce9dc6b82b9caf3d6fbe57c30ed93b35a399c312f08b95f16" exitCode=0 Sep 17 18:38:57.649586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.649195 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" Sep 17 18:38:57.649586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.649219 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" event={"ID":"9c8d7e85-838b-4210-b470-2d1972e63bf6","Type":"ContainerDied","Data":"34cde97c73c3392ce9dc6b82b9caf3d6fbe57c30ed93b35a399c312f08b95f16"} Sep 17 18:38:57.649586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.649295 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc" event={"ID":"9c8d7e85-838b-4210-b470-2d1972e63bf6","Type":"ContainerDied","Data":"caf145813e071b319f089e04afd3ec1819e5664c9b47208e785bb8f5cac60e7b"} Sep 17 18:38:57.649586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.649328 2564 scope.go:117] "RemoveContainer" containerID="34cde97c73c3392ce9dc6b82b9caf3d6fbe57c30ed93b35a399c312f08b95f16" Sep 17 18:38:57.650847 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.650827 2564 generic.go:356] "Generic (PLEG): container finished" podID="16834e97-f84f-48af-b796-607dcef66fb4" containerID="f06fcfdfba05af0f8471e73e03ead05a435653d9090c1b8003995493cf11201b" exitCode=0 Sep 17 18:38:57.650918 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.650884 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" event={"ID":"16834e97-f84f-48af-b796-607dcef66fb4","Type":"ContainerDied","Data":"f06fcfdfba05af0f8471e73e03ead05a435653d9090c1b8003995493cf11201b"} Sep 17 18:38:57.650966 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.650927 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" event={"ID":"16834e97-f84f-48af-b796-607dcef66fb4","Type":"ContainerDied","Data":"aacb2a34db6b652b1429b39d905ffeb1e9e28ac3e2c871c62481847c32369233"} Sep 17 18:38:57.650966 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.650886 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm" Sep 17 18:38:57.659641 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.659621 2564 scope.go:117] "RemoveContainer" containerID="34cde97c73c3392ce9dc6b82b9caf3d6fbe57c30ed93b35a399c312f08b95f16" Sep 17 18:38:57.659946 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:38:57.659927 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34cde97c73c3392ce9dc6b82b9caf3d6fbe57c30ed93b35a399c312f08b95f16\": container with ID starting with 34cde97c73c3392ce9dc6b82b9caf3d6fbe57c30ed93b35a399c312f08b95f16 not found: ID does not exist" containerID="34cde97c73c3392ce9dc6b82b9caf3d6fbe57c30ed93b35a399c312f08b95f16" Sep 17 18:38:57.660006 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.659954 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34cde97c73c3392ce9dc6b82b9caf3d6fbe57c30ed93b35a399c312f08b95f16"} err="failed to get container status \"34cde97c73c3392ce9dc6b82b9caf3d6fbe57c30ed93b35a399c312f08b95f16\": rpc error: code = NotFound desc = could not find container \"34cde97c73c3392ce9dc6b82b9caf3d6fbe57c30ed93b35a399c312f08b95f16\": container with ID starting with 34cde97c73c3392ce9dc6b82b9caf3d6fbe57c30ed93b35a399c312f08b95f16 not found: ID does not exist" Sep 17 18:38:57.660006 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.659971 2564 scope.go:117] "RemoveContainer" containerID="f06fcfdfba05af0f8471e73e03ead05a435653d9090c1b8003995493cf11201b" Sep 17 18:38:57.668288 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.668269 2564 scope.go:117] "RemoveContainer" containerID="f06fcfdfba05af0f8471e73e03ead05a435653d9090c1b8003995493cf11201b" Sep 17 18:38:57.668568 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:38:57.668551 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06fcfdfba05af0f8471e73e03ead05a435653d9090c1b8003995493cf11201b\": container with ID starting with f06fcfdfba05af0f8471e73e03ead05a435653d9090c1b8003995493cf11201b not found: ID does not exist" containerID="f06fcfdfba05af0f8471e73e03ead05a435653d9090c1b8003995493cf11201b" Sep 17 18:38:57.668641 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.668574 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06fcfdfba05af0f8471e73e03ead05a435653d9090c1b8003995493cf11201b"} err="failed to get container status \"f06fcfdfba05af0f8471e73e03ead05a435653d9090c1b8003995493cf11201b\": rpc error: code = NotFound desc = could not find container \"f06fcfdfba05af0f8471e73e03ead05a435653d9090c1b8003995493cf11201b\": container with ID starting with f06fcfdfba05af0f8471e73e03ead05a435653d9090c1b8003995493cf11201b not found: ID does not exist" Sep 17 18:38:57.697313 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.697292 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm"] Sep 17 18:38:57.703956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.703900 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-95a92b-6c876cbd66-rpwcm"] Sep 17 18:38:57.745941 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.745919 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc"] Sep 17 18:38:57.763710 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:57.763687 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-95a92b-78db98d6d7-x4xkc"] Sep 17 18:38:59.451164 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:59.451127 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16834e97-f84f-48af-b796-607dcef66fb4" path="/var/lib/kubelet/pods/16834e97-f84f-48af-b796-607dcef66fb4/volumes" Sep 17 18:38:59.451577 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:38:59.451470 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c8d7e85-838b-4210-b470-2d1972e63bf6" path="/var/lib/kubelet/pods/9c8d7e85-838b-4210-b470-2d1972e63bf6/volumes" Sep 17 18:39:04.683005 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.682970 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p"] Sep 17 18:39:04.683458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683331 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerName="istio-proxy" Sep 17 18:39:04.683458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683348 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerName="istio-proxy" Sep 17 18:39:04.683458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683363 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16834e97-f84f-48af-b796-607dcef66fb4" containerName="payload-pre-processing" Sep 17 18:39:04.683458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683371 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="16834e97-f84f-48af-b796-607dcef66fb4" containerName="payload-pre-processing" Sep 17 18:39:04.683458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683388 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a60bada-5417-408a-8e07-8da36aa92a46" containerName="payload-pre-processing" Sep 17 18:39:04.683458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683396 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a60bada-5417-408a-8e07-8da36aa92a46" containerName="payload-pre-processing" Sep 17 18:39:04.683458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683418 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e8195b-04eb-4c7e-8b2c-7c6472ae702d" containerName="payload-pre-processing" Sep 17 18:39:04.683458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683426 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e8195b-04eb-4c7e-8b2c-7c6472ae702d" containerName="payload-pre-processing" Sep 17 18:39:04.683458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683437 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerName="istio-proxy" Sep 17 18:39:04.683458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683445 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerName="istio-proxy" Sep 17 18:39:04.683458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683454 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6" containerName="revoke-keys" Sep 17 18:39:04.683458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683462 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6" containerName="revoke-keys" Sep 17 18:39:04.683930 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683474 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c8d7e85-838b-4210-b470-2d1972e63bf6" containerName="payload-processing" Sep 17 18:39:04.683930 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683483 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c8d7e85-838b-4210-b470-2d1972e63bf6" containerName="payload-processing" Sep 17 18:39:04.683930 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683591 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5efdb99d-9a7d-4e1b-8268-1b2ea183eeb6" containerName="revoke-keys" Sep 17 18:39:04.683930 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683602 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerName="istio-proxy" Sep 17 18:39:04.683930 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683611 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e8195b-04eb-4c7e-8b2c-7c6472ae702d" containerName="payload-pre-processing" Sep 17 18:39:04.683930 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683623 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a60bada-5417-408a-8e07-8da36aa92a46" containerName="payload-pre-processing" Sep 17 18:39:04.683930 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683631 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerName="istio-proxy" Sep 17 18:39:04.683930 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683640 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerName="istio-proxy" Sep 17 18:39:04.683930 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683648 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="16834e97-f84f-48af-b796-607dcef66fb4" containerName="payload-pre-processing" Sep 17 18:39:04.683930 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683667 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerName="istio-proxy" Sep 17 18:39:04.683930 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.683675 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c8d7e85-838b-4210-b470-2d1972e63bf6" containerName="payload-processing" Sep 17 18:39:04.686878 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.686860 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p" Sep 17 18:39:04.689496 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.689479 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 18:39:04.689645 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.689479 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\"" Sep 17 18:39:04.695771 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.695743 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p"] Sep 17 18:39:04.836767 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.836736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c78a5d6-3ab6-46a2-b100-a115447ce8e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p\" (UID: \"1c78a5d6-3ab6-46a2-b100-a115447ce8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p" Sep 17 18:39:04.937956 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.937883 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c78a5d6-3ab6-46a2-b100-a115447ce8e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p\" (UID: \"1c78a5d6-3ab6-46a2-b100-a115447ce8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p" Sep 17 18:39:04.938463 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.938441 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c78a5d6-3ab6-46a2-b100-a115447ce8e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p\" (UID: \"1c78a5d6-3ab6-46a2-b100-a115447ce8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p" Sep 17 18:39:04.998802 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:04.998777 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p" Sep 17 18:39:05.278867 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:05.278841 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p"] Sep 17 18:39:05.279821 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:39:05.279792 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c78a5d6_3ab6_46a2_b100_a115447ce8e6.slice/crio-a789d66f4aa1c770f3f70e991f9c26f32d41eb4d4064804f5bb2b7a9f8edcfcf WatchSource:0}: Error finding container a789d66f4aa1c770f3f70e991f9c26f32d41eb4d4064804f5bb2b7a9f8edcfcf: Status 404 returned error can't find the container with id a789d66f4aa1c770f3f70e991f9c26f32d41eb4d4064804f5bb2b7a9f8edcfcf Sep 17 18:39:05.701887 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:05.701862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p" event={"ID":"1c78a5d6-3ab6-46a2-b100-a115447ce8e6","Type":"ContainerStarted","Data":"416e77fc721fee0c4aee857cd790acb57736f63809260a2dd2cd3290826f9ce5"} Sep 17 18:39:05.702188 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:05.701909 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p" event={"ID":"1c78a5d6-3ab6-46a2-b100-a115447ce8e6","Type":"ContainerStarted","Data":"a789d66f4aa1c770f3f70e991f9c26f32d41eb4d4064804f5bb2b7a9f8edcfcf"} Sep 17 18:39:06.709448 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:06.709410 2564 generic.go:356] "Generic (PLEG): container finished" podID="1c78a5d6-3ab6-46a2-b100-a115447ce8e6" containerID="416e77fc721fee0c4aee857cd790acb57736f63809260a2dd2cd3290826f9ce5" exitCode=0 Sep 17 18:39:06.709812 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:06.709493 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p" event={"ID":"1c78a5d6-3ab6-46a2-b100-a115447ce8e6","Type":"ContainerDied","Data":"416e77fc721fee0c4aee857cd790acb57736f63809260a2dd2cd3290826f9ce5"} Sep 17 18:39:07.848262 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:07.848240 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p" Sep 17 18:39:07.961375 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:07.961349 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c78a5d6-3ab6-46a2-b100-a115447ce8e6-maas-api-service-ca\") pod \"1c78a5d6-3ab6-46a2-b100-a115447ce8e6\" (UID: \"1c78a5d6-3ab6-46a2-b100-a115447ce8e6\") " Sep 17 18:39:07.961699 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:07.961679 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c78a5d6-3ab6-46a2-b100-a115447ce8e6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c78a5d6-3ab6-46a2-b100-a115447ce8e6" (UID: "1c78a5d6-3ab6-46a2-b100-a115447ce8e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:39:08.062638 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:08.062616 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c78a5d6-3ab6-46a2-b100-a115447ce8e6-maas-api-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:39:08.723493 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:08.723453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p" event={"ID":"1c78a5d6-3ab6-46a2-b100-a115447ce8e6","Type":"ContainerDied","Data":"a789d66f4aa1c770f3f70e991f9c26f32d41eb4d4064804f5bb2b7a9f8edcfcf"} Sep 17 18:39:08.723694 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:08.723516 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a789d66f4aa1c770f3f70e991f9c26f32d41eb4d4064804f5bb2b7a9f8edcfcf" Sep 17 18:39:08.723694 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:08.723473 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p" Sep 17 18:39:15.380204 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.380168 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv"] Sep 17 18:39:15.380620 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.380460 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" podUID="589dcc22-437e-4bbe-964d-a943c6d7c9ec" containerName="payload-processing" containerID="cri-o://bfdeb41e9141bb6c7613ccc1ba1276c9088a84300d4ba99bf714d2942b9b750e" gracePeriod=30 Sep 17 18:39:15.602811 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.602782 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p"] Sep 17 18:39:15.607192 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.607156 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d81d7c-kbf7p"] Sep 17 18:39:15.639684 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.639625 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" Sep 17 18:39:15.764913 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.764870 2564 generic.go:356] "Generic (PLEG): container finished" podID="589dcc22-437e-4bbe-964d-a943c6d7c9ec" containerID="bfdeb41e9141bb6c7613ccc1ba1276c9088a84300d4ba99bf714d2942b9b750e" exitCode=0 Sep 17 18:39:15.764913 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.764913 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" Sep 17 18:39:15.765153 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.764911 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" event={"ID":"589dcc22-437e-4bbe-964d-a943c6d7c9ec","Type":"ContainerDied","Data":"bfdeb41e9141bb6c7613ccc1ba1276c9088a84300d4ba99bf714d2942b9b750e"} Sep 17 18:39:15.765153 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.764978 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv" event={"ID":"589dcc22-437e-4bbe-964d-a943c6d7c9ec","Type":"ContainerDied","Data":"0ec5433c3b54e5a80d7ef18458ec827bcdcd44a0cf37f7c0633b4ccb9a4c4217"} Sep 17 18:39:15.765153 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.765010 2564 scope.go:117] "RemoveContainer" containerID="bfdeb41e9141bb6c7613ccc1ba1276c9088a84300d4ba99bf714d2942b9b750e" Sep 17 18:39:15.774259 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.774241 2564 scope.go:117] "RemoveContainer" containerID="bfdeb41e9141bb6c7613ccc1ba1276c9088a84300d4ba99bf714d2942b9b750e" Sep 17 18:39:15.774555 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:39:15.774519 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdeb41e9141bb6c7613ccc1ba1276c9088a84300d4ba99bf714d2942b9b750e\": container with ID starting with bfdeb41e9141bb6c7613ccc1ba1276c9088a84300d4ba99bf714d2942b9b750e not found: ID does not exist" containerID="bfdeb41e9141bb6c7613ccc1ba1276c9088a84300d4ba99bf714d2942b9b750e" Sep 17 18:39:15.774605 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.774566 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdeb41e9141bb6c7613ccc1ba1276c9088a84300d4ba99bf714d2942b9b750e"} err="failed to get container status \"bfdeb41e9141bb6c7613ccc1ba1276c9088a84300d4ba99bf714d2942b9b750e\": rpc error: code = NotFound desc = could not find container \"bfdeb41e9141bb6c7613ccc1ba1276c9088a84300d4ba99bf714d2942b9b750e\": container with ID starting with bfdeb41e9141bb6c7613ccc1ba1276c9088a84300d4ba99bf714d2942b9b750e not found: ID does not exist" Sep 17 18:39:15.826196 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.826175 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589dcc22-437e-4bbe-964d-a943c6d7c9ec-plugins-config-volume\") pod \"589dcc22-437e-4bbe-964d-a943c6d7c9ec\" (UID: \"589dcc22-437e-4bbe-964d-a943c6d7c9ec\") " Sep 17 18:39:15.826313 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.826205 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2dgr\" (UniqueName: \"kubernetes.io/projected/589dcc22-437e-4bbe-964d-a943c6d7c9ec-kube-api-access-l2dgr\") pod \"589dcc22-437e-4bbe-964d-a943c6d7c9ec\" (UID: \"589dcc22-437e-4bbe-964d-a943c6d7c9ec\") " Sep 17 18:39:15.826625 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.826604 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589dcc22-437e-4bbe-964d-a943c6d7c9ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "589dcc22-437e-4bbe-964d-a943c6d7c9ec" (UID: "589dcc22-437e-4bbe-964d-a943c6d7c9ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:39:15.828421 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.828400 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589dcc22-437e-4bbe-964d-a943c6d7c9ec-kube-api-access-l2dgr" (OuterVolumeSpecName: "kube-api-access-l2dgr") pod "589dcc22-437e-4bbe-964d-a943c6d7c9ec" (UID: "589dcc22-437e-4bbe-964d-a943c6d7c9ec"). InnerVolumeSpecName "kube-api-access-l2dgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:39:15.927135 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.927079 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589dcc22-437e-4bbe-964d-a943c6d7c9ec-plugins-config-volume\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:39:15.927135 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:15.927105 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2dgr\" (UniqueName: \"kubernetes.io/projected/589dcc22-437e-4bbe-964d-a943c6d7c9ec-kube-api-access-l2dgr\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:39:16.090337 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:16.090302 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv"] Sep 17 18:39:16.092921 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:16.092895 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d81d7c-56db445699-n7cxv"] Sep 17 18:39:17.451476 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:17.451435 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c78a5d6-3ab6-46a2-b100-a115447ce8e6" path="/var/lib/kubelet/pods/1c78a5d6-3ab6-46a2-b100-a115447ce8e6/volumes" Sep 17 18:39:17.451923 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:17.451890 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="589dcc22-437e-4bbe-964d-a943c6d7c9ec" path="/var/lib/kubelet/pods/589dcc22-437e-4bbe-964d-a943c6d7c9ec/volumes" Sep 17 18:39:22.917586 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:22.917557 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s"] Sep 17 18:39:22.917946 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:22.917772 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" containerID="cri-o://bf088340b337ad9a56b4f5a2e663fa637b5f36ef7b6f0a4c94ebce48c86f12c9" gracePeriod=30 Sep 17 18:39:28.165456 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.165436 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:39:28.224955 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.224886 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-workload-certs\") pod \"efdef7ba-ba6b-464b-88d6-618baeab04bc\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " Sep 17 18:39:28.224955 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.224941 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-credential-socket\") pod \"efdef7ba-ba6b-464b-88d6-618baeab04bc\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " Sep 17 18:39:28.225144 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.224967 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-data\") pod \"efdef7ba-ba6b-464b-88d6-618baeab04bc\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " Sep 17 18:39:28.225144 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.224993 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvxck\" (UniqueName: \"kubernetes.io/projected/efdef7ba-ba6b-464b-88d6-618baeab04bc-kube-api-access-wvxck\") pod \"efdef7ba-ba6b-464b-88d6-618baeab04bc\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " Sep 17 18:39:28.225144 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.225055 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-envoy\") pod \"efdef7ba-ba6b-464b-88d6-618baeab04bc\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " Sep 17 18:39:28.225144 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.225084 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-token\") pod \"efdef7ba-ba6b-464b-88d6-618baeab04bc\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " Sep 17 18:39:28.225358 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.225146 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-podinfo\") pod \"efdef7ba-ba6b-464b-88d6-618baeab04bc\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " Sep 17 18:39:28.225358 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.225181 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-workload-socket\") pod \"efdef7ba-ba6b-464b-88d6-618baeab04bc\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " Sep 17 18:39:28.225358 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.225181 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "efdef7ba-ba6b-464b-88d6-618baeab04bc" (UID: "efdef7ba-ba6b-464b-88d6-618baeab04bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:39:28.225358 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.225217 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efdef7ba-ba6b-464b-88d6-618baeab04bc-istiod-ca-cert\") pod \"efdef7ba-ba6b-464b-88d6-618baeab04bc\" (UID: \"efdef7ba-ba6b-464b-88d6-618baeab04bc\") " Sep 17 18:39:28.225358 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.225232 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "efdef7ba-ba6b-464b-88d6-618baeab04bc" (UID: "efdef7ba-ba6b-464b-88d6-618baeab04bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:39:28.225358 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.225319 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "efdef7ba-ba6b-464b-88d6-618baeab04bc" (UID: "efdef7ba-ba6b-464b-88d6-618baeab04bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:39:28.226116 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.225940 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "efdef7ba-ba6b-464b-88d6-618baeab04bc" (UID: "efdef7ba-ba6b-464b-88d6-618baeab04bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:39:28.226116 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.226040 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efdef7ba-ba6b-464b-88d6-618baeab04bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "efdef7ba-ba6b-464b-88d6-618baeab04bc" (UID: "efdef7ba-ba6b-464b-88d6-618baeab04bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:39:28.226431 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.226411 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:39:28.226492 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.226448 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:39:28.226492 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.226465 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:39:28.228860 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.228000 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "efdef7ba-ba6b-464b-88d6-618baeab04bc" (UID: "efdef7ba-ba6b-464b-88d6-618baeab04bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:39:28.228860 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.228290 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "efdef7ba-ba6b-464b-88d6-618baeab04bc" (UID: "efdef7ba-ba6b-464b-88d6-618baeab04bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 18:39:28.228860 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.229449 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efdef7ba-ba6b-464b-88d6-618baeab04bc-kube-api-access-wvxck" (OuterVolumeSpecName: "kube-api-access-wvxck") pod "efdef7ba-ba6b-464b-88d6-618baeab04bc" (UID: "efdef7ba-ba6b-464b-88d6-618baeab04bc"). InnerVolumeSpecName "kube-api-access-wvxck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:39:28.234598 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.234574 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "efdef7ba-ba6b-464b-88d6-618baeab04bc" (UID: "efdef7ba-ba6b-464b-88d6-618baeab04bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:39:28.327689 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.327667 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efdef7ba-ba6b-464b-88d6-618baeab04bc-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:39:28.327689 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.327687 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvxck\" (UniqueName: \"kubernetes.io/projected/efdef7ba-ba6b-464b-88d6-618baeab04bc-kube-api-access-wvxck\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:39:28.327820 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.327698 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:39:28.327820 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.327705 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:39:28.327820 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.327713 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efdef7ba-ba6b-464b-88d6-618baeab04bc-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:39:28.327820 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.327721 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efdef7ba-ba6b-464b-88d6-618baeab04bc-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:39:28.852357 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.852324 2564 generic.go:356] "Generic (PLEG): container finished" podID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerID="bf088340b337ad9a56b4f5a2e663fa637b5f36ef7b6f0a4c94ebce48c86f12c9" exitCode=0 Sep 17 18:39:28.852583 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.852366 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" event={"ID":"efdef7ba-ba6b-464b-88d6-618baeab04bc","Type":"ContainerDied","Data":"bf088340b337ad9a56b4f5a2e663fa637b5f36ef7b6f0a4c94ebce48c86f12c9"} Sep 17 18:39:28.852583 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.852409 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" Sep 17 18:39:28.852583 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.852443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s" event={"ID":"efdef7ba-ba6b-464b-88d6-618baeab04bc","Type":"ContainerDied","Data":"0700ac560b1bc5ddeeb3f962eea1191039a241dc38bf58bb4f6cd86af0018ba5"} Sep 17 18:39:28.852583 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.852469 2564 scope.go:117] "RemoveContainer" containerID="bf088340b337ad9a56b4f5a2e663fa637b5f36ef7b6f0a4c94ebce48c86f12c9" Sep 17 18:39:28.863245 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.863228 2564 scope.go:117] "RemoveContainer" containerID="b9fbf131888229e4e6f6b4050901320b06816a03fe3e1ca60b5cf3092e17a21a" Sep 17 18:39:28.872284 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.872076 2564 scope.go:117] "RemoveContainer" containerID="bf088340b337ad9a56b4f5a2e663fa637b5f36ef7b6f0a4c94ebce48c86f12c9" Sep 17 18:39:28.872637 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:39:28.872610 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf088340b337ad9a56b4f5a2e663fa637b5f36ef7b6f0a4c94ebce48c86f12c9\": container with ID starting with bf088340b337ad9a56b4f5a2e663fa637b5f36ef7b6f0a4c94ebce48c86f12c9 not found: ID does not exist" containerID="bf088340b337ad9a56b4f5a2e663fa637b5f36ef7b6f0a4c94ebce48c86f12c9" Sep 17 18:39:28.872724 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.872644 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf088340b337ad9a56b4f5a2e663fa637b5f36ef7b6f0a4c94ebce48c86f12c9"} err="failed to get container status \"bf088340b337ad9a56b4f5a2e663fa637b5f36ef7b6f0a4c94ebce48c86f12c9\": rpc error: code = NotFound desc = could not find container \"bf088340b337ad9a56b4f5a2e663fa637b5f36ef7b6f0a4c94ebce48c86f12c9\": container with ID starting with bf088340b337ad9a56b4f5a2e663fa637b5f36ef7b6f0a4c94ebce48c86f12c9 not found: ID does not exist" Sep 17 18:39:28.872724 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.872663 2564 scope.go:117] "RemoveContainer" containerID="b9fbf131888229e4e6f6b4050901320b06816a03fe3e1ca60b5cf3092e17a21a" Sep 17 18:39:28.872982 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:39:28.872962 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fbf131888229e4e6f6b4050901320b06816a03fe3e1ca60b5cf3092e17a21a\": container with ID starting with b9fbf131888229e4e6f6b4050901320b06816a03fe3e1ca60b5cf3092e17a21a not found: ID does not exist" containerID="b9fbf131888229e4e6f6b4050901320b06816a03fe3e1ca60b5cf3092e17a21a" Sep 17 18:39:28.873038 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.872986 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fbf131888229e4e6f6b4050901320b06816a03fe3e1ca60b5cf3092e17a21a"} err="failed to get container status \"b9fbf131888229e4e6f6b4050901320b06816a03fe3e1ca60b5cf3092e17a21a\": rpc error: code = NotFound desc = could not find container \"b9fbf131888229e4e6f6b4050901320b06816a03fe3e1ca60b5cf3092e17a21a\": container with ID starting with b9fbf131888229e4e6f6b4050901320b06816a03fe3e1ca60b5cf3092e17a21a not found: ID does not exist" Sep 17 18:39:28.875344 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.875320 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s"] Sep 17 18:39:28.900000 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:28.899972 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d81d7c-openshift-default-66d75f9c4d-jj59s"] Sep 17 18:39:29.450478 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:29.450448 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" path="/var/lib/kubelet/pods/efdef7ba-ba6b-464b-88d6-618baeab04bc/volumes" Sep 17 18:39:55.940591 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:39:55.940556 2564 scope.go:117] "RemoveContainer" containerID="c79fc153b65aabdcc35c0651b8641acb9008d7fb7387fbbc2dfc7c1b14c9a96d" Sep 17 18:42:56.000056 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:42:56.000019 2564 scope.go:117] "RemoveContainer" containerID="56d39e8352a4e5dd1ded22aef7d0886999cadf2e8f6c81da092cec65006611a7" Sep 17 18:43:56.019685 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:43:56.019645 2564 scope.go:117] "RemoveContainer" containerID="2996f902c72cb92208c23db24d0246393864bf66b928a3ddc330246f22bcbfe5" Sep 17 18:45:00.137997 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.137911 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827845-lbd5h"] Sep 17 18:45:00.138404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.138282 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" Sep 17 18:45:00.138404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.138294 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" Sep 17 18:45:00.138404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.138317 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerName="istio-proxy" Sep 17 18:45:00.138404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.138323 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="efcab250-f7ce-4d5f-9c1d-d62d6243ead6" containerName="istio-proxy" Sep 17 18:45:00.138404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.138337 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="589dcc22-437e-4bbe-964d-a943c6d7c9ec" containerName="payload-processing" Sep 17 18:45:00.138404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.138344 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="589dcc22-437e-4bbe-964d-a943c6d7c9ec" containerName="payload-processing" Sep 17 18:45:00.138404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.138351 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c78a5d6-3ab6-46a2-b100-a115447ce8e6" containerName="revoke-keys" Sep 17 18:45:00.138404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.138357 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c78a5d6-3ab6-46a2-b100-a115447ce8e6" containerName="revoke-keys" Sep 17 18:45:00.138404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.138364 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerName="istio-proxy" Sep 17 18:45:00.138404 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.138369 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a084ce3-4df0-474a-8bd2-d99e26fe8109" containerName="istio-proxy" Sep 17 18:45:00.138767 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.138442 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" Sep 17 18:45:00.138767 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.138450 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c78a5d6-3ab6-46a2-b100-a115447ce8e6" containerName="revoke-keys" Sep 17 18:45:00.138767 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.138459 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="589dcc22-437e-4bbe-964d-a943c6d7c9ec" containerName="payload-processing" Sep 17 18:45:00.141711 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.141692 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827845-lbd5h" Sep 17 18:45:00.145055 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.145025 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\"" Sep 17 18:45:00.153834 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.153809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827845-lbd5h" Sep 17 18:45:00.154351 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.154328 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827845-lbd5h"] Sep 17 18:45:00.437285 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.437234 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827845-lbd5h"] Sep 17 18:45:00.439049 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:45:00.439024 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0aff46_9898_4630_84f2_ec89d27a1eae.slice/crio-f8b6b29b9e625752030b249da607315800a4d4e5a947caf9402df9bae5e730cc WatchSource:0}: Error finding container f8b6b29b9e625752030b249da607315800a4d4e5a947caf9402df9bae5e730cc: Status 404 returned error can't find the container with id f8b6b29b9e625752030b249da607315800a4d4e5a947caf9402df9bae5e730cc Sep 17 18:45:00.683414 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:00.683379 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827845-lbd5h" event={"ID":"5b0aff46-9898-4630-84f2-ec89d27a1eae","Type":"ContainerStarted","Data":"f8b6b29b9e625752030b249da607315800a4d4e5a947caf9402df9bae5e730cc"} Sep 17 18:45:01.691305 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:01.691274 2564 generic.go:356] "Generic (PLEG): container finished" podID="5b0aff46-9898-4630-84f2-ec89d27a1eae" containerID="3c51dd6981bdc8d4c36729553977a5ceda6672929b102539716683b7ff404018" exitCode=0 Sep 17 18:45:01.691698 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:01.691310 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827845-lbd5h" event={"ID":"5b0aff46-9898-4630-84f2-ec89d27a1eae","Type":"ContainerDied","Data":"3c51dd6981bdc8d4c36729553977a5ceda6672929b102539716683b7ff404018"} Sep 17 18:45:02.832447 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:02.832424 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827845-lbd5h" Sep 17 18:45:03.704174 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:03.704138 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827845-lbd5h" event={"ID":"5b0aff46-9898-4630-84f2-ec89d27a1eae","Type":"ContainerDied","Data":"f8b6b29b9e625752030b249da607315800a4d4e5a947caf9402df9bae5e730cc"} Sep 17 18:45:03.704312 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:03.704177 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827845-lbd5h" Sep 17 18:45:03.704312 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:03.704193 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8b6b29b9e625752030b249da607315800a4d4e5a947caf9402df9bae5e730cc" Sep 17 18:45:11.303914 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.303868 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc"] Sep 17 18:45:11.304517 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.304395 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" Sep 17 18:45:11.304517 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.304418 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" Sep 17 18:45:11.304517 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.304443 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0aff46-9898-4630-84f2-ec89d27a1eae" containerName="cleanup" Sep 17 18:45:11.304517 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.304452 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0aff46-9898-4630-84f2-ec89d27a1eae" containerName="cleanup" Sep 17 18:45:11.304778 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.304576 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b0aff46-9898-4630-84f2-ec89d27a1eae" containerName="cleanup" Sep 17 18:45:11.304778 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.304592 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="efdef7ba-ba6b-464b-88d6-618baeab04bc" containerName="istio-proxy" Sep 17 18:45:11.307831 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.307809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.311669 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.311640 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sqjt5\"" Sep 17 18:45:11.311808 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.311647 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 18:45:11.311808 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.311698 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 18:45:11.311808 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.311705 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 17 18:45:11.311808 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.311716 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 18:45:11.318149 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.318124 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc"] Sep 17 18:45:11.459094 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.459065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/494899f1-d58f-4be0-9a00-d8107cf9206b-openshift-service-ca\") pod \"maas-api-5ffb5cf8dc-ttwkc\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.459094 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.459095 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/494899f1-d58f-4be0-9a00-d8107cf9206b-maas-api-tls\") pod \"maas-api-5ffb5cf8dc-ttwkc\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.459283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.459166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/494899f1-d58f-4be0-9a00-d8107cf9206b-metrics-certs\") pod \"maas-api-5ffb5cf8dc-ttwkc\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.459283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.459203 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6phks\" (UniqueName: \"kubernetes.io/projected/494899f1-d58f-4be0-9a00-d8107cf9206b-kube-api-access-6phks\") pod \"maas-api-5ffb5cf8dc-ttwkc\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.560097 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.560026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6phks\" (UniqueName: \"kubernetes.io/projected/494899f1-d58f-4be0-9a00-d8107cf9206b-kube-api-access-6phks\") pod \"maas-api-5ffb5cf8dc-ttwkc\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.560256 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.560108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/494899f1-d58f-4be0-9a00-d8107cf9206b-openshift-service-ca\") pod \"maas-api-5ffb5cf8dc-ttwkc\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.560256 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.560135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/494899f1-d58f-4be0-9a00-d8107cf9206b-maas-api-tls\") pod \"maas-api-5ffb5cf8dc-ttwkc\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.560256 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.560201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/494899f1-d58f-4be0-9a00-d8107cf9206b-metrics-certs\") pod \"maas-api-5ffb5cf8dc-ttwkc\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.560837 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.560812 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/494899f1-d58f-4be0-9a00-d8107cf9206b-openshift-service-ca\") pod \"maas-api-5ffb5cf8dc-ttwkc\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.562785 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.562760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/494899f1-d58f-4be0-9a00-d8107cf9206b-maas-api-tls\") pod \"maas-api-5ffb5cf8dc-ttwkc\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.562902 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.562881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/494899f1-d58f-4be0-9a00-d8107cf9206b-metrics-certs\") pod \"maas-api-5ffb5cf8dc-ttwkc\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.568227 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.568209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6phks\" (UniqueName: \"kubernetes.io/projected/494899f1-d58f-4be0-9a00-d8107cf9206b-kube-api-access-6phks\") pod \"maas-api-5ffb5cf8dc-ttwkc\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.621186 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.621166 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:11.896248 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.896224 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc"] Sep 17 18:45:11.897226 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:45:11.897201 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494899f1_d58f_4be0_9a00_d8107cf9206b.slice/crio-4b04960191235042ea502a9d1e588ea2ccd9ea4011f5dd6bfd39b6ae584dcb91 WatchSource:0}: Error finding container 4b04960191235042ea502a9d1e588ea2ccd9ea4011f5dd6bfd39b6ae584dcb91: Status 404 returned error can't find the container with id 4b04960191235042ea502a9d1e588ea2ccd9ea4011f5dd6bfd39b6ae584dcb91 Sep 17 18:45:11.898664 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:11.898646 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:45:12.754999 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:12.754952 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" event={"ID":"494899f1-d58f-4be0-9a00-d8107cf9206b","Type":"ContainerStarted","Data":"4b04960191235042ea502a9d1e588ea2ccd9ea4011f5dd6bfd39b6ae584dcb91"} Sep 17 18:45:15.774331 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:15.774292 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" event={"ID":"494899f1-d58f-4be0-9a00-d8107cf9206b","Type":"ContainerStarted","Data":"b411f988b0d96d3048f48cda79b0561ab136c7d5475adfe85debb7995d2d8dcb"} Sep 17 18:45:15.774753 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:15.774406 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:15.792916 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:15.792865 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" podStartSLOduration=1.720734279 podStartE2EDuration="4.792851186s" totalImagesPullingTime="3.072116907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:15.790945028 +0000 UTC m=+2540.801132179" watchObservedRunningTime="2026-09-17 18:45:15.792851186 +0000 UTC m=+2540.803038327" Sep 17 18:45:18.781929 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:18.781889 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc"] Sep 17 18:45:18.782415 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:18.782067 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" podUID="494899f1-d58f-4be0-9a00-d8107cf9206b" containerName="maas-api" containerID="cri-o://b411f988b0d96d3048f48cda79b0561ab136c7d5475adfe85debb7995d2d8dcb" gracePeriod=30 Sep 17 18:45:18.815377 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:18.815353 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp"] Sep 17 18:45:18.818595 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:18.818573 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:18.829241 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:18.829217 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp"] Sep 17 18:45:18.921132 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:18.921096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35e2f488-51be-4b4d-950d-16e6864ae8ca-openshift-service-ca\") pod \"maas-api-776dd778bc-hxqwp\" (UID: \"35e2f488-51be-4b4d-950d-16e6864ae8ca\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:18.921283 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:18.921163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltwdj\" (UniqueName: \"kubernetes.io/projected/35e2f488-51be-4b4d-950d-16e6864ae8ca-kube-api-access-ltwdj\") pod \"maas-api-776dd778bc-hxqwp\" (UID: \"35e2f488-51be-4b4d-950d-16e6864ae8ca\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:18.921341 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:18.921270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35e2f488-51be-4b4d-950d-16e6864ae8ca-metrics-certs\") pod \"maas-api-776dd778bc-hxqwp\" (UID: \"35e2f488-51be-4b4d-950d-16e6864ae8ca\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:18.921381 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:18.921365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e2f488-51be-4b4d-950d-16e6864ae8ca-maas-api-tls\") pod \"maas-api-776dd778bc-hxqwp\" (UID: \"35e2f488-51be-4b4d-950d-16e6864ae8ca\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:19.022084 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.022062 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:19.022227 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.022206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e2f488-51be-4b4d-950d-16e6864ae8ca-maas-api-tls\") pod \"maas-api-776dd778bc-hxqwp\" (UID: \"35e2f488-51be-4b4d-950d-16e6864ae8ca\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:19.022272 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.022244 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35e2f488-51be-4b4d-950d-16e6864ae8ca-openshift-service-ca\") pod \"maas-api-776dd778bc-hxqwp\" (UID: \"35e2f488-51be-4b4d-950d-16e6864ae8ca\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:19.022305 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.022287 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltwdj\" (UniqueName: \"kubernetes.io/projected/35e2f488-51be-4b4d-950d-16e6864ae8ca-kube-api-access-ltwdj\") pod \"maas-api-776dd778bc-hxqwp\" (UID: \"35e2f488-51be-4b4d-950d-16e6864ae8ca\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:19.022339 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.022333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35e2f488-51be-4b4d-950d-16e6864ae8ca-metrics-certs\") pod \"maas-api-776dd778bc-hxqwp\" (UID: \"35e2f488-51be-4b4d-950d-16e6864ae8ca\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:19.022963 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.022940 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35e2f488-51be-4b4d-950d-16e6864ae8ca-openshift-service-ca\") pod \"maas-api-776dd778bc-hxqwp\" (UID: \"35e2f488-51be-4b4d-950d-16e6864ae8ca\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:19.024652 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.024636 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35e2f488-51be-4b4d-950d-16e6864ae8ca-metrics-certs\") pod \"maas-api-776dd778bc-hxqwp\" (UID: \"35e2f488-51be-4b4d-950d-16e6864ae8ca\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:19.024791 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.024773 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e2f488-51be-4b4d-950d-16e6864ae8ca-maas-api-tls\") pod \"maas-api-776dd778bc-hxqwp\" (UID: \"35e2f488-51be-4b4d-950d-16e6864ae8ca\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:19.031101 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.031078 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltwdj\" (UniqueName: \"kubernetes.io/projected/35e2f488-51be-4b4d-950d-16e6864ae8ca-kube-api-access-ltwdj\") pod \"maas-api-776dd778bc-hxqwp\" (UID: \"35e2f488-51be-4b4d-950d-16e6864ae8ca\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:19.123000 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.122975 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6phks\" (UniqueName: \"kubernetes.io/projected/494899f1-d58f-4be0-9a00-d8107cf9206b-kube-api-access-6phks\") pod \"494899f1-d58f-4be0-9a00-d8107cf9206b\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " Sep 17 18:45:19.123126 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.123003 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/494899f1-d58f-4be0-9a00-d8107cf9206b-openshift-service-ca\") pod \"494899f1-d58f-4be0-9a00-d8107cf9206b\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " Sep 17 18:45:19.123126 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.123061 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/494899f1-d58f-4be0-9a00-d8107cf9206b-metrics-certs\") pod \"494899f1-d58f-4be0-9a00-d8107cf9206b\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " Sep 17 18:45:19.123126 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.123117 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/494899f1-d58f-4be0-9a00-d8107cf9206b-maas-api-tls\") pod \"494899f1-d58f-4be0-9a00-d8107cf9206b\" (UID: \"494899f1-d58f-4be0-9a00-d8107cf9206b\") " Sep 17 18:45:19.123381 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.123359 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494899f1-d58f-4be0-9a00-d8107cf9206b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "494899f1-d58f-4be0-9a00-d8107cf9206b" (UID: "494899f1-d58f-4be0-9a00-d8107cf9206b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:45:19.125238 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.125216 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494899f1-d58f-4be0-9a00-d8107cf9206b-kube-api-access-6phks" (OuterVolumeSpecName: "kube-api-access-6phks") pod "494899f1-d58f-4be0-9a00-d8107cf9206b" (UID: "494899f1-d58f-4be0-9a00-d8107cf9206b"). InnerVolumeSpecName "kube-api-access-6phks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:45:19.125324 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.125235 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494899f1-d58f-4be0-9a00-d8107cf9206b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "494899f1-d58f-4be0-9a00-d8107cf9206b" (UID: "494899f1-d58f-4be0-9a00-d8107cf9206b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:45:19.125324 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.125249 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494899f1-d58f-4be0-9a00-d8107cf9206b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "494899f1-d58f-4be0-9a00-d8107cf9206b" (UID: "494899f1-d58f-4be0-9a00-d8107cf9206b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:45:19.156580 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.156563 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:19.224170 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.224143 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/494899f1-d58f-4be0-9a00-d8107cf9206b-maas-api-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:45:19.224170 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.224172 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6phks\" (UniqueName: \"kubernetes.io/projected/494899f1-d58f-4be0-9a00-d8107cf9206b-kube-api-access-6phks\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:45:19.224309 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.224183 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/494899f1-d58f-4be0-9a00-d8107cf9206b-openshift-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:45:19.224309 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.224195 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/494899f1-d58f-4be0-9a00-d8107cf9206b-metrics-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Sep 17 18:45:19.430695 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.430672 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp"] Sep 17 18:45:19.432996 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:45:19.432965 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e2f488_51be_4b4d_950d_16e6864ae8ca.slice/crio-b38147b80d6de12ad268374a003e861d2bda4b47f8104465a59590ee7a1ba878 WatchSource:0}: Error finding container b38147b80d6de12ad268374a003e861d2bda4b47f8104465a59590ee7a1ba878: Status 404 returned error can't find the container with id b38147b80d6de12ad268374a003e861d2bda4b47f8104465a59590ee7a1ba878 Sep 17 18:45:19.798952 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.798876 2564 generic.go:356] "Generic (PLEG): container finished" podID="494899f1-d58f-4be0-9a00-d8107cf9206b" containerID="b411f988b0d96d3048f48cda79b0561ab136c7d5475adfe85debb7995d2d8dcb" exitCode=0 Sep 17 18:45:19.799326 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.798943 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" Sep 17 18:45:19.799326 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.798955 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" event={"ID":"494899f1-d58f-4be0-9a00-d8107cf9206b","Type":"ContainerDied","Data":"b411f988b0d96d3048f48cda79b0561ab136c7d5475adfe85debb7995d2d8dcb"} Sep 17 18:45:19.799326 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.799005 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc" event={"ID":"494899f1-d58f-4be0-9a00-d8107cf9206b","Type":"ContainerDied","Data":"4b04960191235042ea502a9d1e588ea2ccd9ea4011f5dd6bfd39b6ae584dcb91"} Sep 17 18:45:19.799326 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.799027 2564 scope.go:117] "RemoveContainer" containerID="b411f988b0d96d3048f48cda79b0561ab136c7d5475adfe85debb7995d2d8dcb" Sep 17 18:45:19.800820 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.800799 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" event={"ID":"35e2f488-51be-4b4d-950d-16e6864ae8ca","Type":"ContainerStarted","Data":"b38147b80d6de12ad268374a003e861d2bda4b47f8104465a59590ee7a1ba878"} Sep 17 18:45:19.809191 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.809172 2564 scope.go:117] "RemoveContainer" containerID="b411f988b0d96d3048f48cda79b0561ab136c7d5475adfe85debb7995d2d8dcb" Sep 17 18:45:19.809482 ip-10-0-138-138 kubenswrapper[2564]: E0917 18:45:19.809458 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b411f988b0d96d3048f48cda79b0561ab136c7d5475adfe85debb7995d2d8dcb\": container with ID starting with b411f988b0d96d3048f48cda79b0561ab136c7d5475adfe85debb7995d2d8dcb not found: ID does not exist" containerID="b411f988b0d96d3048f48cda79b0561ab136c7d5475adfe85debb7995d2d8dcb" Sep 17 18:45:19.809605 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.809489 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b411f988b0d96d3048f48cda79b0561ab136c7d5475adfe85debb7995d2d8dcb"} err="failed to get container status \"b411f988b0d96d3048f48cda79b0561ab136c7d5475adfe85debb7995d2d8dcb\": rpc error: code = NotFound desc = could not find container \"b411f988b0d96d3048f48cda79b0561ab136c7d5475adfe85debb7995d2d8dcb\": container with ID starting with b411f988b0d96d3048f48cda79b0561ab136c7d5475adfe85debb7995d2d8dcb not found: ID does not exist" Sep 17 18:45:19.818829 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.818806 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc"] Sep 17 18:45:19.822704 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:19.822684 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-ttwkc"] Sep 17 18:45:20.808084 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:20.808045 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" event={"ID":"35e2f488-51be-4b4d-950d-16e6864ae8ca","Type":"ContainerStarted","Data":"c172cdaac044268c0fd79b954a7272bcb87dbd25b48e35a9e9ec417c9d43b2f3"} Sep 17 18:45:20.808516 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:20.808118 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:20.826108 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:20.826061 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" podStartSLOduration=2.31533384 podStartE2EDuration="2.826046116s" totalImagesPullingTime="510.712276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:20.825329202 +0000 UTC m=+2545.835516339" watchObservedRunningTime="2026-09-17 18:45:20.826046116 +0000 UTC m=+2545.836233252" Sep 17 18:45:21.451293 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:21.451261 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="494899f1-d58f-4be0-9a00-d8107cf9206b" path="/var/lib/kubelet/pods/494899f1-d58f-4be0-9a00-d8107cf9206b/volumes" Sep 17 18:45:26.820477 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:26.820443 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-776dd778bc-hxqwp" Sep 17 18:45:46.953360 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:46.953331 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-gb5lj_d30ba015-e452-4a82-a227-7bf26d504a93/server/0.log" Sep 17 18:45:47.072393 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:47.072369 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vb689_0df7fffc-37a3-46eb-9767-455a50dfc379/manager/0.log" Sep 17 18:45:47.194564 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:47.194517 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-648c847c44-65nxm_7093f3f1-2b75-40b1-8148-f70d97f35fb2/copy-manifests/0.log" Sep 17 18:45:47.201744 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:47.201724 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-648c847c44-65nxm_7093f3f1-2b75-40b1-8148-f70d97f35fb2/manager/0.log" Sep 17 18:45:47.815639 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:47.815600 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-776dd778bc-hxqwp_35e2f488-51be-4b4d-950d-16e6864ae8ca/maas-api/0.log" Sep 17 18:45:48.037458 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:48.037421 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29827815-vhl6d_d60ed2f1-0e88-49e0-886f-9ffe6f964741/cleanup/0.log" Sep 17 18:45:48.261379 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:48.261280 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29827845-lbd5h_5b0aff46-9898-4630-84f2-ec89d27a1eae/cleanup/0.log" Sep 17 18:45:50.209577 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:50.209543 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw_e74a9357-78d5-4afb-babe-31e5a16973b5/istio-proxy/0.log" Sep 17 18:45:50.332785 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:50.332759 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cmpkf_e49f4b11-b236-44a4-9078-b43c79561b83/discovery/0.log" Sep 17 18:45:51.065124 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:51.065094 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64665745fb-n4rxl_8cc392b6-399b-4293-9c92-afc8dbf7d952/router/0.log" Sep 17 18:45:51.410005 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:51.409934 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-5h4fv_d29f1d01-405e-4022-a3f5-fd6426f28706/main/0.log" Sep 17 18:45:51.523057 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:51.523035 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-6czzx_070c27f8-c599-4724-aae0-2e65aa5a0481/main/0.log" Sep 17 18:45:56.050168 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:45:56.050136 2564 scope.go:117] "RemoveContainer" containerID="416e77fc721fee0c4aee857cd790acb57736f63809260a2dd2cd3290826f9ce5" Sep 17 18:46:01.072906 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:01.072873 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l2fvr_0704f62c-880a-451b-851f-731517d14596/global-pull-secret-syncer/0.log" Sep 17 18:46:01.231732 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:01.231704 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f87tt_21e09d04-55d0-4833-8f9c-ec78ea8beffc/konnectivity-agent/0.log" Sep 17 18:46:01.336343 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:01.336264 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-138.ec2.internal_d18622155b9da2257267bc123b9432af/haproxy/0.log" Sep 17 18:46:08.550180 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:08.550094 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-gq22m_a4a5e89b-2526-495d-8321-fc7825a6e74e/monitoring-plugin/0.log" Sep 17 18:46:08.688200 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:08.688176 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slq6v_df993f22-21e2-4efb-9fe8-9328204d734d/node-exporter/0.log" Sep 17 18:46:08.718595 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:08.718567 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slq6v_df993f22-21e2-4efb-9fe8-9328204d734d/kube-rbac-proxy/0.log" Sep 17 18:46:08.745784 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:08.745765 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slq6v_df993f22-21e2-4efb-9fe8-9328204d734d/init-textfile/0.log" Sep 17 18:46:09.222924 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:09.222895 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-lm6lf_feceb9c7-986d-4b94-b85d-4aa1f42d1d7a/prometheus-operator/0.log" Sep 17 18:46:09.248148 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:09.248125 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-lm6lf_feceb9c7-986d-4b94-b85d-4aa1f42d1d7a/kube-rbac-proxy/0.log" Sep 17 18:46:09.277463 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:09.277436 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-95jxg_7e3b9c9b-e1e6-46ab-879e-b7e3b35538af/prometheus-operator-admission-webhook/0.log" Sep 17 18:46:09.312099 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:09.312079 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68b846668c-m2vs5_ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691/telemeter-client/0.log" Sep 17 18:46:09.334947 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:09.334925 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68b846668c-m2vs5_ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691/reload/0.log" Sep 17 18:46:09.359777 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:09.359756 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68b846668c-m2vs5_ea9b1f01-7ec0-49c6-9ce1-ae85c6fa9691/kube-rbac-proxy/0.log" Sep 17 18:46:09.394962 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:09.394936 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b67466fcd-h7hn9_51a10561-f1f2-4157-a74e-8ff241d6a9e2/thanos-query/0.log" Sep 17 18:46:09.420335 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:09.420303 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b67466fcd-h7hn9_51a10561-f1f2-4157-a74e-8ff241d6a9e2/kube-rbac-proxy-web/0.log" Sep 17 18:46:09.443196 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:09.443182 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b67466fcd-h7hn9_51a10561-f1f2-4157-a74e-8ff241d6a9e2/kube-rbac-proxy/0.log" Sep 17 18:46:09.466539 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:09.466503 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b67466fcd-h7hn9_51a10561-f1f2-4157-a74e-8ff241d6a9e2/prom-label-proxy/0.log" Sep 17 18:46:09.489131 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:09.489067 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b67466fcd-h7hn9_51a10561-f1f2-4157-a74e-8ff241d6a9e2/kube-rbac-proxy-rules/0.log" Sep 17 18:46:09.516408 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:09.516375 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b67466fcd-h7hn9_51a10561-f1f2-4157-a74e-8ff241d6a9e2/kube-rbac-proxy-metrics/0.log" Sep 17 18:46:11.255772 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.255739 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl"] Sep 17 18:46:11.256215 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.256082 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="494899f1-d58f-4be0-9a00-d8107cf9206b" containerName="maas-api" Sep 17 18:46:11.256215 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.256093 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="494899f1-d58f-4be0-9a00-d8107cf9206b" containerName="maas-api" Sep 17 18:46:11.256215 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.256160 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="494899f1-d58f-4be0-9a00-d8107cf9206b" containerName="maas-api" Sep 17 18:46:11.259503 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.259483 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.262584 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.262565 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vmxpt\"/\"default-dockercfg-pgspx\"" Sep 17 18:46:11.263406 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.263389 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vmxpt\"/\"kube-root-ca.crt\"" Sep 17 18:46:11.263461 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.263389 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vmxpt\"/\"openshift-service-ca.crt\"" Sep 17 18:46:11.274060 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.274039 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl"] Sep 17 18:46:11.324128 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.324099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d5f68d9-0967-4e24-85bd-85322243f176-podres\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.324224 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.324138 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d5f68d9-0967-4e24-85bd-85322243f176-proc\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.324224 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.324166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d5f68d9-0967-4e24-85bd-85322243f176-sys\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.324224 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.324203 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk457\" (UniqueName: \"kubernetes.io/projected/3d5f68d9-0967-4e24-85bd-85322243f176-kube-api-access-pk457\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.324356 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.324241 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d5f68d9-0967-4e24-85bd-85322243f176-lib-modules\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.425276 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.425248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d5f68d9-0967-4e24-85bd-85322243f176-proc\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.425379 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.425284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d5f68d9-0967-4e24-85bd-85322243f176-sys\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.425379 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.425309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk457\" (UniqueName: \"kubernetes.io/projected/3d5f68d9-0967-4e24-85bd-85322243f176-kube-api-access-pk457\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.425379 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.425346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d5f68d9-0967-4e24-85bd-85322243f176-lib-modules\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.425379 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.425361 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d5f68d9-0967-4e24-85bd-85322243f176-proc\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.425379 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.425368 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d5f68d9-0967-4e24-85bd-85322243f176-sys\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.425646 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.425471 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d5f68d9-0967-4e24-85bd-85322243f176-podres\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.425646 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.425506 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d5f68d9-0967-4e24-85bd-85322243f176-lib-modules\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.425646 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.425605 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d5f68d9-0967-4e24-85bd-85322243f176-podres\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.434086 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.434062 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk457\" (UniqueName: \"kubernetes.io/projected/3d5f68d9-0967-4e24-85bd-85322243f176-kube-api-access-pk457\") pod \"perf-node-gather-daemonset-wgvfl\" (UID: \"3d5f68d9-0967-4e24-85bd-85322243f176\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.570221 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.570200 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:11.844448 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:11.844420 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl"] Sep 17 18:46:11.845987 ip-10-0-138-138 kubenswrapper[2564]: W0917 18:46:11.845960 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3d5f68d9_0967_4e24_85bd_85322243f176.slice/crio-015322e1956006681e3f03160f8e5b034d532ed2cd3d9e22564a6b6d114d3aff WatchSource:0}: Error finding container 015322e1956006681e3f03160f8e5b034d532ed2cd3d9e22564a6b6d114d3aff: Status 404 returned error can't find the container with id 015322e1956006681e3f03160f8e5b034d532ed2cd3d9e22564a6b6d114d3aff Sep 17 18:46:12.108732 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:12.108651 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" event={"ID":"3d5f68d9-0967-4e24-85bd-85322243f176","Type":"ContainerStarted","Data":"3751a7b7785f0255da0b7c29747597ff2a3d5c0bc0ba69b9994f42f79be7b295"} Sep 17 18:46:12.108732 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:12.108709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" event={"ID":"3d5f68d9-0967-4e24-85bd-85322243f176","Type":"ContainerStarted","Data":"015322e1956006681e3f03160f8e5b034d532ed2cd3d9e22564a6b6d114d3aff"} Sep 17 18:46:12.108907 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:12.108809 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:12.125891 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:12.125847 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" podStartSLOduration=1.125830974 podStartE2EDuration="1.125830974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:12.123997061 +0000 UTC m=+2597.134184211" watchObservedRunningTime="2026-09-17 18:46:12.125830974 +0000 UTC m=+2597.136018111" Sep 17 18:46:14.245002 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:14.244956 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9zglp_b8385b17-3d85-46f2-abbc-f92dd8cc9357/dns/0.log" Sep 17 18:46:14.279227 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:14.279200 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9zglp_b8385b17-3d85-46f2-abbc-f92dd8cc9357/kube-rbac-proxy/0.log" Sep 17 18:46:14.364356 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:14.364332 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h9q2n_b73423cd-ef53-471d-8b77-c72793a394b8/dns-node-resolver/0.log" Sep 17 18:46:15.052056 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:15.052016 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-d97cbd56c-r9w88_a2d962b4-8f76-4dc7-be4a-e9ed7aa53a43/registry/0.log" Sep 17 18:46:15.079694 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:15.079661 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8hr8c_91d98768-2d1f-4172-a9c7-f782aa9056ac/node-ca/0.log" Sep 17 18:46:16.146836 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:16.146807 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw_e74a9357-78d5-4afb-babe-31e5a16973b5/istio-proxy/0.log" Sep 17 18:46:16.578082 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:16.578049 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cmpkf_e49f4b11-b236-44a4-9078-b43c79561b83/discovery/0.log" Sep 17 18:46:16.785716 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:16.785679 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64665745fb-n4rxl_8cc392b6-399b-4293-9c92-afc8dbf7d952/router/0.log" Sep 17 18:46:17.765292 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:17.765205 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lcwnb_d4e5b8f7-05b4-4cf0-bb6e-8230dcfbad8d/serve-healthcheck-canary/0.log" Sep 17 18:46:18.125627 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:18.125605 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-wgvfl" Sep 17 18:46:18.301884 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:18.301859 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-9xz92_c3b9adda-9178-46dd-885b-9ffa426d7e03/insights-operator/0.log" Sep 17 18:46:18.302137 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:18.302121 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-9xz92_c3b9adda-9178-46dd-885b-9ffa426d7e03/insights-operator/1.log" Sep 17 18:46:18.324191 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:18.324172 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-22vs2_99fc2ce8-e335-4833-81b3-9c99d1897474/kube-rbac-proxy/0.log" Sep 17 18:46:18.349798 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:18.349781 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-22vs2_99fc2ce8-e335-4833-81b3-9c99d1897474/exporter/0.log" Sep 17 18:46:18.373762 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:18.373742 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-22vs2_99fc2ce8-e335-4833-81b3-9c99d1897474/extractor/0.log" Sep 17 18:46:21.584202 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:21.584173 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-gb5lj_d30ba015-e452-4a82-a227-7bf26d504a93/server/0.log" Sep 17 18:46:21.673244 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:21.673220 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vb689_0df7fffc-37a3-46eb-9767-455a50dfc379/manager/0.log" Sep 17 18:46:21.699788 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:21.699741 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-648c847c44-65nxm_7093f3f1-2b75-40b1-8148-f70d97f35fb2/manager/0.log" Sep 17 18:46:21.731870 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:21.731855 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-648c847c44-65nxm_7093f3f1-2b75-40b1-8148-f70d97f35fb2/copy-manifests/0.log" Sep 17 18:46:22.392707 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:22.392684 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw_e74a9357-78d5-4afb-babe-31e5a16973b5/istio-proxy/0.log" Sep 17 18:46:22.799379 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:22.799308 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cmpkf_e49f4b11-b236-44a4-9078-b43c79561b83/discovery/0.log" Sep 17 18:46:22.947850 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:22.947826 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64665745fb-n4rxl_8cc392b6-399b-4293-9c92-afc8dbf7d952/router/0.log" Sep 17 18:46:23.251399 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:23.251372 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-b8b8795dc-bx2tg_c63ec5b2-c4f6-4900-aef5-0dac872ee565/manager/0.log" Sep 17 18:46:24.431723 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:24.431697 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-86ljw_e74a9357-78d5-4afb-babe-31e5a16973b5/istio-proxy/0.log" Sep 17 18:46:24.854949 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:24.854922 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cmpkf_e49f4b11-b236-44a4-9078-b43c79561b83/discovery/0.log" Sep 17 18:46:25.033422 ip-10-0-138-138 kubenswrapper[2564]: I0917 18:46:25.033399 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64665745fb-n4rxl_8cc392b6-399b-4293-9c92-afc8dbf7d952/router/0.log"