Aug 01 00:28:31.086347 ip-10-0-128-195 systemd[1]: Starting Kubernetes Kubelet... Aug 01 00:28:31.520982 ip-10-0-128-195 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 01 00:28:31.520982 ip-10-0-128-195 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 01 00:28:31.520982 ip-10-0-128-195 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 01 00:28:31.520982 ip-10-0-128-195 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 01 00:28:31.520982 ip-10-0-128-195 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 01 00:28:31.526172 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.525999 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 01 00:28:31.531092 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531078 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:28:31.531092 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531092 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531095 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531099 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531105 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531109 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531112 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531116 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531118 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531121 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531124 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531126 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531129 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531132 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531135 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531137 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531140 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531143 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531145 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531148 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:28:31.531163 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531150 2563 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531153 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531155 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531158 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531160 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531163 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531165 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531168 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531171 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531174 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531183 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531186 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531189 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531192 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531194 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531197 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531199 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531202 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531204 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531207 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:28:31.531600 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531209 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531211 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531214 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531216 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531219 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531222 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531224 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531226 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531230 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531233 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531236 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531239 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531242 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531244 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531247 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531250 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531253 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531255 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531258 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531261 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:28:31.532076 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531264 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531267 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531269 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531272 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531274 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531278 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531280 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531283 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531285 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531288 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531291 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531294 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531297 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531300 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531302 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531305 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531307 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531310 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531313 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:28:31.532590 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531315 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531317 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531320 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531323 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531325 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531329 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531331 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531334 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531337 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531339 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531342 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531344 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531346 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531349 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531352 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531354 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531356 2563 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531359 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531361 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.531364 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:28:31.533067 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532010 2563 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532016 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532019 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532022 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532024 2563 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532027 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532029 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532032 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532035 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532037 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532040 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532042 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532045 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532048 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532051 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532067 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532070 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532073 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532075 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532079 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:28:31.533559 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532083 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532087 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532090 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532093 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532097 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532101 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532105 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532109 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532112 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532115 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532118 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532121 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532124 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532127 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532130 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532132 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532135 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532137 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532140 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:28:31.534047 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532142 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532145 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532147 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532150 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532152 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532155 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532157 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532159 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532162 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532165 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532168 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532171 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532175 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532179 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532182 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532186 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532188 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532191 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532193 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:28:31.534518 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532196 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532198 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532201 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532203 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532206 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532209 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532212 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532214 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532216 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532219 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532221 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532224 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532226 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532229 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532231 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532234 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532236 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532238 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532241 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532243 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:28:31.534972 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532246 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532248 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532258 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532262 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532264 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532267 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532270 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532273 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532275 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532278 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532280 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532282 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532285 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532305 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532308 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532310 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532313 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532315 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532318 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532321 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:28:31.535484 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532323 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532680 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532684 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532688 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532691 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532694 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532696 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532699 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532702 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532704 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532707 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532709 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532712 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532714 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532723 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532726 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532729 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532732 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532734 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:28:31.535956 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532737 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532739 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532742 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532745 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532747 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532750 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532752 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532755 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532757 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532760 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532762 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532765 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532768 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532770 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532773 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532776 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532779 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532781 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532783 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532786 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:28:31.536456 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532789 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532791 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532794 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532797 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532799 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532802 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532804 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532807 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532811 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532815 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532820 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532823 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532827 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532830 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532833 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532835 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532838 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532841 2563 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532843 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532846 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:28:31.536967 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532848 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532851 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532853 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532856 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532858 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532861 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532863 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532866 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532868 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532870 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532881 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532884 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532886 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532889 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532892 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532895 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532899 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532901 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532904 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532907 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:28:31.537466 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532911 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532914 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532916 2563 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532919 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532922 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532925 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532927 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532930 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532934 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532937 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532939 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532942 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532945 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532947 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532950 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532952 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532955 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532957 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532960 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532962 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:28:31.537955 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.532965 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533034 2563 flags.go:64] FLAG: --address="0.0.0.0" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533042 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533048 2563 flags.go:64] FLAG: --anonymous-auth="true" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533053 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533070 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533074 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533078 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533083 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533086 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533089 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533095 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533098 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533101 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533105 2563 flags.go:64] FLAG: --cgroup-root="" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533107 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533111 2563 flags.go:64] FLAG: --client-ca-file="" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533113 2563 flags.go:64] FLAG: --cloud-provider="external" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533116 2563 flags.go:64] FLAG: --cluster-dns="[]" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533120 2563 flags.go:64] FLAG: --cluster-domain="" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533125 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533128 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533131 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533134 2563 flags.go:64] FLAG: --container-log-max-files="5" Aug 01 00:28:31.538499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533138 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533141 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533144 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533147 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533150 2563 flags.go:64] FLAG: --contention-profiling="false" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533153 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533156 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533159 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533162 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533166 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533168 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533171 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533174 2563 flags.go:64] FLAG: --enable-load-reader="false" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533178 2563 flags.go:64] FLAG: --enable-server="true" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533180 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533185 2563 flags.go:64] FLAG: --event-burst="100" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533188 2563 flags.go:64] FLAG: --event-qps="50" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533191 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533194 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533198 2563 flags.go:64] FLAG: --eviction-hard="" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533201 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533204 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533207 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533211 2563 flags.go:64] FLAG: --eviction-soft="" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533214 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 01 00:28:31.539104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533217 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533219 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533222 2563 flags.go:64] FLAG: --experimental-mounter-path="" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533227 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533229 2563 flags.go:64] FLAG: --fail-swap-on="true" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533232 2563 flags.go:64] FLAG: --feature-gates="" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533236 2563 flags.go:64] FLAG: --file-check-frequency="20s" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533239 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533242 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533245 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533247 2563 flags.go:64] FLAG: --healthz-port="10248" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533250 2563 flags.go:64] FLAG: --help="false" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533253 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-128-195.ec2.internal" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533256 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533259 2563 flags.go:64] FLAG: --http-check-frequency="20s" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533262 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533265 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533268 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533271 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533274 2563 flags.go:64] FLAG: --image-service-endpoint="" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533277 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533279 2563 flags.go:64] FLAG: --kube-api-burst="100" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533282 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533285 2563 flags.go:64] FLAG: --kube-api-qps="50" Aug 01 00:28:31.539742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533288 2563 flags.go:64] FLAG: --kube-reserved="" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533291 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533295 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533298 2563 flags.go:64] FLAG: --kubelet-cgroups="" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533301 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533303 2563 flags.go:64] FLAG: --lock-file="" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533306 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533310 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533312 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533318 2563 flags.go:64] FLAG: --log-json-split-stream="false" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533321 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533325 2563 flags.go:64] FLAG: --log-text-split-stream="false" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533327 2563 flags.go:64] FLAG: --logging-format="text" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533330 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533334 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533336 2563 flags.go:64] FLAG: --manifest-url="" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533339 2563 flags.go:64] FLAG: --manifest-url-header="" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533343 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533346 2563 flags.go:64] FLAG: --max-open-files="1000000" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533350 2563 flags.go:64] FLAG: --max-pods="110" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533353 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533356 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533359 2563 flags.go:64] FLAG: --memory-manager-policy="None" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533362 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533365 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 01 00:28:31.540374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533368 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533371 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533378 2563 flags.go:64] FLAG: --node-status-max-images="50" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533381 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533384 2563 flags.go:64] FLAG: --oom-score-adj="-999" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533387 2563 flags.go:64] FLAG: --pod-cidr="" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533390 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533396 2563 flags.go:64] FLAG: --pod-manifest-path="" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533399 2563 flags.go:64] FLAG: --pod-max-pids="-1" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533402 2563 flags.go:64] FLAG: --pods-per-core="0" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533407 2563 flags.go:64] FLAG: --port="10250" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533411 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533413 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04b49cd32a0532519" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533417 2563 flags.go:64] FLAG: --qos-reserved="" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533419 2563 flags.go:64] FLAG: --read-only-port="10255" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533426 2563 flags.go:64] FLAG: --register-node="true" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533430 2563 flags.go:64] FLAG: --register-with-taints="" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533434 2563 flags.go:64] FLAG: --registry-burst="10" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533438 2563 flags.go:64] FLAG: --registry-qps="5" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533441 2563 flags.go:64] FLAG: --reserved-cpus="" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533443 2563 flags.go:64] FLAG: --reserved-memory="" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533447 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533450 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533453 2563 flags.go:64] FLAG: --rotate-certificates="false" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533456 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 01 00:28:31.541414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533459 2563 flags.go:64] FLAG: --runonce="false" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533462 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533465 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533481 2563 flags.go:64] FLAG: --seccomp-default="false" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533484 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533487 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533490 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533493 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533496 2563 flags.go:64] FLAG: --storage-driver-password="root" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533498 2563 flags.go:64] FLAG: --storage-driver-secure="false" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533501 2563 flags.go:64] FLAG: --storage-driver-table="stats" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533504 2563 flags.go:64] FLAG: --storage-driver-user="root" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533507 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533509 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533512 2563 flags.go:64] FLAG: --system-cgroups="" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533515 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533520 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533524 2563 flags.go:64] FLAG: --tls-cert-file="" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533527 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533530 2563 flags.go:64] FLAG: --tls-min-version="" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533533 2563 flags.go:64] FLAG: --tls-private-key-file="" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533535 2563 flags.go:64] FLAG: --topology-manager-policy="none" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533538 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533546 2563 flags.go:64] FLAG: --topology-manager-scope="container" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533549 2563 flags.go:64] FLAG: --v="2" Aug 01 00:28:31.542094 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533554 2563 flags.go:64] FLAG: --version="false" Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533558 2563 flags.go:64] FLAG: --vmodule="" Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533562 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.533565 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533694 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533697 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533700 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533703 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533706 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533708 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533711 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533713 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533716 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533719 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533721 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533724 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533726 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533729 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533731 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533734 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:28:31.543031 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533737 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533739 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533741 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533745 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533751 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533754 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533756 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533759 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533761 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533764 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533766 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533770 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533773 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533776 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533779 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533781 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533784 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533786 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533789 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:28:31.544003 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533791 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533794 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533796 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533798 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533801 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533803 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533806 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533808 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533811 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533813 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533817 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533821 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533823 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533826 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533829 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533832 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533834 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533838 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533840 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:28:31.544883 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533843 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533846 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533848 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533851 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533853 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533856 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533867 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533869 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533872 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533874 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533877 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533879 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533882 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533884 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533892 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533895 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533897 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533900 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533902 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:28:31.545482 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533904 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533908 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533911 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533914 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533917 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533920 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533923 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533925 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533928 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533930 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533933 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533937 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533940 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533943 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533946 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533948 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533951 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533953 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533956 2563 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533960 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:28:31.546032 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533963 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533965 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533968 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533970 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533973 2563 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.533976 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.534827 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.543639 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.543657 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.543683 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.543692 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543776 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543783 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543788 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543793 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:28:31.546852 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543798 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543803 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543807 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543812 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543816 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543820 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543825 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543829 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543833 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543837 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543842 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543846 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543850 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543854 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543859 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543863 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543867 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543871 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543875 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:28:31.547387 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543882 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543888 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543892 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543897 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543901 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543905 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543914 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543918 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543923 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543927 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543931 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543936 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543940 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543944 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543948 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543952 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543957 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543961 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543965 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543969 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:28:31.547829 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543973 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543977 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543981 2563 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543985 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543989 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543993 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.543997 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544001 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544007 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544013 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544017 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544021 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544026 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544031 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544035 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544040 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544045 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544050 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544073 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:28:31.548460 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544079 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544085 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544089 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544094 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544098 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544102 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544107 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544111 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544116 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544120 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544124 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544129 2563 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544133 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544137 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544141 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544145 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544149 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544156 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544161 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:28:31.548946 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544165 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544170 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544174 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544179 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544184 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544188 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544193 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544197 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544201 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544206 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544210 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544215 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544219 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544223 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544228 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544233 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544237 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:28:31.549444 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544241 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.544254 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544452 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544461 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544465 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544469 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544473 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544477 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544482 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544487 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544490 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544495 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544499 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544503 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544508 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:28:31.549938 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544512 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544516 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544520 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544524 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544528 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544532 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544537 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544541 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544546 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544550 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544554 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544559 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544564 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544568 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544572 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544577 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544581 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544586 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544590 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544595 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:28:31.550322 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544599 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544603 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544607 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544611 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544615 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544619 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544626 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544632 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544637 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544642 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544646 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544650 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544654 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544659 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544663 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544667 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544671 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544675 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544680 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:28:31.550838 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544684 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544688 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544694 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544699 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544703 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544708 2563 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544712 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544716 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544720 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544725 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544730 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544733 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544738 2563 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544742 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544746 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544750 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544754 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544758 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544762 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544766 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:28:31.551316 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544771 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544775 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544779 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544783 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544787 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544791 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544795 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544799 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544803 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544810 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544815 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544819 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544824 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544829 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544833 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544837 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544841 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544845 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544850 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:28:31.551814 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544854 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:28:31.552315 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544858 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:28:31.552315 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544862 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:28:31.552315 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544867 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:28:31.552315 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544871 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:28:31.552315 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544875 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:28:31.552315 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544879 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:28:31.552315 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:31.544883 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:28:31.552315 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.544890 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 01 00:28:31.552315 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.545039 2563 server.go:962] "Client rotation is on, will bootstrap in background" Aug 01 00:28:31.552315 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.547975 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 01 00:28:31.552315 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.549813 2563 server.go:1019] "Starting client certificate rotation" Aug 01 00:28:31.552315 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.549909 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 01 00:28:31.552315 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.549948 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 01 00:28:31.552910 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.552897 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 01 00:28:31.556278 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.556259 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 01 00:28:31.568602 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.568580 2563 log.go:25] "Validated CRI v1 runtime API" Aug 01 00:28:31.571095 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.571081 2563 log.go:25] "Validated CRI v1 image API" Aug 01 00:28:31.571882 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.571870 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 01 00:28:31.576720 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.576693 2563 fs.go:135] Filesystem UUIDs: map[29e44fc1-286f-4098-91b7-7aeb25722d96:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 970900c6-8c60-4abc-83ad-79adb96f4152:/dev/nvme0n1p4] Aug 01 00:28:31.576786 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.576720 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 01 00:28:31.579430 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.579414 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 01 00:28:31.581988 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.581888 2563 manager.go:216] Machine: {Timestamp:2026-08-01 00:28:31.580535895 +0000 UTC m=+0.369257716 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3196066 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eb5436544bb4fcc378b5c0a44eeb4 SystemUUID:ec2eb543-6544-bb4f-cc37-8b5c0a44eeb4 BootID:3d4c5206-5396-49d2-bec9-560f68012d36 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6b:4f:54:fd:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6b:4f:54:fd:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:48:26:16:e4:43 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 01 00:28:31.581988 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.581981 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 01 00:28:31.582131 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.582049 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 01 00:28:31.585538 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.585515 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 01 00:28:31.585673 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.585541 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-195.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 01 00:28:31.586440 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.586430 2563 topology_manager.go:138] "Creating topology manager with none policy" Aug 01 00:28:31.586472 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.586442 2563 container_manager_linux.go:309] "Creating device plugin manager" Aug 01 00:28:31.586472 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.586449 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 01 00:28:31.586472 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.586463 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 01 00:28:31.588366 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.588356 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 01 00:28:31.593641 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.593630 2563 state_mem.go:36] "Initialized new in-memory state store" Aug 01 00:28:31.593735 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.593726 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 01 00:28:31.593772 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.593753 2563 kubelet.go:486] "Attempting to sync node with API server" Aug 01 00:28:31.593772 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.593769 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 01 00:28:31.593821 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.593782 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 01 00:28:31.593821 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.593791 2563 kubelet.go:398] "Adding apiserver pod source" Aug 01 00:28:31.593821 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.593806 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 01 00:28:31.594105 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.594095 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 01 00:28:31.594151 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.594111 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 01 00:28:31.597191 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.597176 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 01 00:28:31.600156 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.600135 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 01 00:28:31.600233 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.600177 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 01 00:28:31.601015 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.600997 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 01 00:28:31.601105 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.601022 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 01 00:28:31.601105 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.601033 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 01 00:28:31.601105 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.601042 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 01 00:28:31.601105 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.601079 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 01 00:28:31.601105 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.601089 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 01 00:28:31.601105 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.601100 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 01 00:28:31.601105 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.601108 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 01 00:28:31.601296 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.601119 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 01 00:28:31.601296 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.601128 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 01 00:28:31.601296 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.601158 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 01 00:28:31.601296 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.601167 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 01 00:28:31.602192 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.602173 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 01 00:28:31.602192 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.602190 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 01 00:28:31.605350 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.605335 2563 server.go:1268] "Started kubelet" Aug 01 00:28:31.605444 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.605417 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 01 00:28:31.605509 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.605420 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 01 00:28:31.605509 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.605499 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 01 00:28:31.606245 ip-10-0-128-195 systemd[1]: Started Kubernetes Kubelet. Aug 01 00:28:31.607244 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.606996 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-195.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 01 00:28:31.607244 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.606991 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 01 00:28:31.607364 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.607263 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-195.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 01 00:28:31.608862 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.608820 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 01 00:28:31.610105 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.610091 2563 server.go:310] "Adding debug handlers to kubelet server" Aug 01 00:28:31.613842 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.612666 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-195.ec2.internal.18c786e8c70659d9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-195.ec2.internal,UID:ip-10-0-128-195.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-195.ec2.internal,},FirstTimestamp:2026-08-01 00:28:31.605307865 +0000 UTC m=+0.394029683,LastTimestamp:2026-08-01 00:28:31.605307865 +0000 UTC m=+0.394029683,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-195.ec2.internal,}" Aug 01 00:28:31.613921 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.613867 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 01 00:28:31.617382 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.617361 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 01 00:28:31.617382 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.617370 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 01 00:28:31.617556 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.617538 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:31.618114 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.618096 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 01 00:28:31.618114 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.618099 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 01 00:28:31.618227 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.618125 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 01 00:28:31.618278 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.618231 2563 reconstruct.go:97] "Volume reconstruction finished" Aug 01 00:28:31.618278 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.618240 2563 reconciler.go:29] "Reconciler: start to sync state" Aug 01 00:28:31.618358 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.618350 2563 factory.go:153] Registering CRI-O factory Aug 01 00:28:31.618412 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.618362 2563 factory.go:223] Registration of the crio container factory successfully Aug 01 00:28:31.618412 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.618404 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 01 00:28:31.618492 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.618413 2563 factory.go:55] Registering systemd factory Aug 01 00:28:31.618492 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.618420 2563 factory.go:223] Registration of the systemd container factory successfully Aug 01 00:28:31.618492 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.618442 2563 factory.go:103] Registering Raw factory Aug 01 00:28:31.618492 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.618452 2563 manager.go:1196] Started watching for new ooms in manager Aug 01 00:28:31.618780 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.618767 2563 manager.go:349] Starting recovery of all containers Aug 01 00:28:31.622625 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.622595 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-195.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 01 00:28:31.622705 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.622682 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 01 00:28:31.626456 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.626426 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6ztfq" Aug 01 00:28:31.628924 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.628802 2563 manager.go:354] Recovery completed Aug 01 00:28:31.632279 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.632263 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6ztfq" Aug 01 00:28:31.633572 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.633557 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:28:31.635395 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.635380 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:28:31.635465 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.635404 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:28:31.635465 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.635414 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:28:31.635899 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.635886 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 01 00:28:31.635899 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.635899 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 01 00:28:31.636014 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.635916 2563 state_mem.go:36] "Initialized new in-memory state store" Aug 01 00:28:31.636014 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.635899 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:28:31.636665 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.636652 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:28:31.636726 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.636674 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:28:31.636726 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.636686 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:28:31.637276 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.637209 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-195.ec2.internal.18c786e8c8d16c68 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-195.ec2.internal,UID:ip-10-0-128-195.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-195.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-195.ec2.internal,},FirstTimestamp:2026-08-01 00:28:31.63539364 +0000 UTC m=+0.424115456,LastTimestamp:2026-08-01 00:28:31.63539364 +0000 UTC m=+0.424115456,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-195.ec2.internal,}" Aug 01 00:28:31.638129 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.638117 2563 policy_none.go:49] "None policy: Start" Aug 01 00:28:31.638177 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.638141 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 01 00:28:31.638177 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.638152 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 01 00:28:31.640103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.640091 2563 policy_none.go:47] "Start" Aug 01 00:28:31.676608 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.676595 2563 manager.go:337] "Starting Device Plugin manager" Aug 01 00:28:31.676693 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.676622 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 01 00:28:31.676693 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.676631 2563 server.go:87] "Starting device plugin registration server" Aug 01 00:28:31.676866 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.676854 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 01 00:28:31.676910 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.676868 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 01 00:28:31.676968 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.676951 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 01 00:28:31.677073 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.677038 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 01 00:28:31.677161 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.677087 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 01 00:28:31.677519 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.677492 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 01 00:28:31.677590 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.677544 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:31.720624 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.720585 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 01 00:28:31.722680 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.721670 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 01 00:28:31.722680 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.721723 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 01 00:28:31.722680 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.721749 2563 kubelet.go:2448] "Starting kubelet main sync loop" Aug 01 00:28:31.722680 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.721788 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 01 00:28:31.723685 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.723664 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:28:31.777488 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.777446 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:28:31.778219 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.778207 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:28:31.778277 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.778232 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:28:31.778277 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.778246 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:28:31.778277 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.778271 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-195.ec2.internal" Aug 01 00:28:31.790209 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.790191 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-195.ec2.internal" Aug 01 00:28:31.790209 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.790209 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-195.ec2.internal\": node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:31.807986 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.807968 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:31.822363 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.822345 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-195.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal"] Aug 01 00:28:31.822413 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.822400 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:28:31.823640 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.823622 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:28:31.823727 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.823645 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:28:31.823727 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.823656 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:28:31.825843 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.825830 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:28:31.825945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.825931 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-195.ec2.internal" Aug 01 00:28:31.825991 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.825961 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:28:31.826496 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.826475 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:28:31.826566 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.826498 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:28:31.826566 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.826513 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:28:31.826627 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.826566 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:28:31.826627 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.826589 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:28:31.826627 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.826603 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:28:31.828748 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.828732 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal" Aug 01 00:28:31.828833 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.828760 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:28:31.829394 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.829380 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:28:31.829462 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.829404 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:28:31.829462 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.829414 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:28:31.853748 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.853730 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-195.ec2.internal\" not found" node="ip-10-0-128-195.ec2.internal" Aug 01 00:28:31.857962 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.857949 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-195.ec2.internal\" not found" node="ip-10-0-128-195.ec2.internal" Aug 01 00:28:31.908494 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:31.908470 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:31.919666 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.919646 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc8c8050d8f35c84ce46fb25d5f43d30-config\") pod \"kube-apiserver-proxy-ip-10-0-128-195.ec2.internal\" (UID: \"bc8c8050d8f35c84ce46fb25d5f43d30\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-195.ec2.internal" Aug 01 00:28:31.919754 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.919678 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63b803daad13bf5020111e921e186d74-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal\" (UID: \"63b803daad13bf5020111e921e186d74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal" Aug 01 00:28:31.919754 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:31.919699 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63b803daad13bf5020111e921e186d74-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal\" (UID: \"63b803daad13bf5020111e921e186d74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal" Aug 01 00:28:32.008822 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:32.008799 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:32.020128 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.020111 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc8c8050d8f35c84ce46fb25d5f43d30-config\") pod \"kube-apiserver-proxy-ip-10-0-128-195.ec2.internal\" (UID: \"bc8c8050d8f35c84ce46fb25d5f43d30\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-195.ec2.internal" Aug 01 00:28:32.020185 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.020133 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63b803daad13bf5020111e921e186d74-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal\" (UID: \"63b803daad13bf5020111e921e186d74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal" Aug 01 00:28:32.020185 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.020149 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63b803daad13bf5020111e921e186d74-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal\" (UID: \"63b803daad13bf5020111e921e186d74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal" Aug 01 00:28:32.020185 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.020179 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63b803daad13bf5020111e921e186d74-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal\" (UID: \"63b803daad13bf5020111e921e186d74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal" Aug 01 00:28:32.020331 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.020197 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc8c8050d8f35c84ce46fb25d5f43d30-config\") pod \"kube-apiserver-proxy-ip-10-0-128-195.ec2.internal\" (UID: \"bc8c8050d8f35c84ce46fb25d5f43d30\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-195.ec2.internal" Aug 01 00:28:32.020331 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.020221 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63b803daad13bf5020111e921e186d74-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal\" (UID: \"63b803daad13bf5020111e921e186d74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal" Aug 01 00:28:32.109570 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:32.109510 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:32.156039 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.156015 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-195.ec2.internal" Aug 01 00:28:32.161363 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.161090 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal" Aug 01 00:28:32.210467 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:32.210320 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:32.310814 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:32.310781 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:32.415933 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:32.415866 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:32.519006 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:32.518973 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:32.549582 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.549560 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 01 00:28:32.549966 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.549692 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 01 00:28:32.619499 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:32.619476 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:32.638437 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.638406 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-31 00:23:32 +0000 UTC" deadline="2028-03-05 05:57:02.728795104 +0000 UTC" Aug 01 00:28:32.638437 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.638434 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13973h28m30.090364319s" Aug 01 00:28:32.719916 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:32.719839 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:32.740111 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:32.740075 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8c8050d8f35c84ce46fb25d5f43d30.slice/crio-7fb4b7d8ab704ad4795e931d2c55dc507f70af04d3cc2a65594e944d14108dfc WatchSource:0}: Error finding container 7fb4b7d8ab704ad4795e931d2c55dc507f70af04d3cc2a65594e944d14108dfc: Status 404 returned error can't find the container with id 7fb4b7d8ab704ad4795e931d2c55dc507f70af04d3cc2a65594e944d14108dfc Aug 01 00:28:32.744876 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.744862 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 00:28:32.822166 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:32.822134 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:32.924329 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:32.924294 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:32.969580 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:32.969553 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:28:33.003520 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:33.003497 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b803daad13bf5020111e921e186d74.slice/crio-06a4308425a2c2cdde7204771a909101b72d5b202a5b8268ecb743da0651927d WatchSource:0}: Error finding container 06a4308425a2c2cdde7204771a909101b72d5b202a5b8268ecb743da0651927d: Status 404 returned error can't find the container with id 06a4308425a2c2cdde7204771a909101b72d5b202a5b8268ecb743da0651927d Aug 01 00:28:33.020189 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.020167 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:28:33.025372 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:33.025354 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-195.ec2.internal\" not found" Aug 01 00:28:33.036348 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.036325 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:28:33.118697 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.118661 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal" Aug 01 00:28:33.132802 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.132784 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 01 00:28:33.133892 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.133878 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-195.ec2.internal" Aug 01 00:28:33.143764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.143748 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 01 00:28:33.595276 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.595244 2563 apiserver.go:52] "Watching apiserver" Aug 01 00:28:33.603613 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.603589 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 01 00:28:33.604031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.604002 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal","openshift-multus/multus-additional-cni-plugins-q2wgc","openshift-multus/multus-gx8gw","openshift-multus/network-metrics-daemon-wkj56","openshift-network-operator/iptables-alerter-jl7q2","kube-system/global-pull-secret-syncer-nlw9l","kube-system/konnectivity-agent-sbb29","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7","openshift-cluster-node-tuning-operator/tuned-69l6n","openshift-image-registry/node-ca-8w9d6","openshift-network-diagnostics/network-check-target-z87nn","openshift-ovn-kubernetes/ovnkube-node-b54j8","kube-system/kube-apiserver-proxy-ip-10-0-128-195.ec2.internal","openshift-dns/node-resolver-5b74f"] Aug 01 00:28:33.606685 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.606667 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8w9d6" Aug 01 00:28:33.609187 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.609053 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:33.609187 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:33.609145 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:33.609187 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.609172 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 01 00:28:33.609428 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.609252 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dv5t2\"" Aug 01 00:28:33.609478 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.609421 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 01 00:28:33.609674 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.609657 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 01 00:28:33.611426 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.611399 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5b74f" Aug 01 00:28:33.613913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.613895 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:33.614019 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:33.613983 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:33.614381 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.614360 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x6dbx\"" Aug 01 00:28:33.614465 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.614419 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 01 00:28:33.614575 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.614559 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 01 00:28:33.616237 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.616218 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jl7q2" Aug 01 00:28:33.617729 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.617713 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 01 00:28:33.618953 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.618741 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j8gqb\"" Aug 01 00:28:33.618953 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.618758 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:28:33.618953 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.618790 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 01 00:28:33.618953 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.618802 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.619268 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.619025 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 01 00:28:33.621429 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.621409 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.621926 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.621881 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 01 00:28:33.622013 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.621884 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-z2nt4\"" Aug 01 00:28:33.622091 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.622052 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 01 00:28:33.622221 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.622201 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 01 00:28:33.622221 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.622217 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 01 00:28:33.622370 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.622231 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 01 00:28:33.623889 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.623641 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 01 00:28:33.623889 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.623697 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:33.623889 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:33.623741 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:33.624074 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.624043 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:28:33.624311 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.624296 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mbxgq\"" Aug 01 00:28:33.628008 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.627989 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sbb29" Aug 01 00:28:33.628818 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.628771 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fee8d179-f489-4751-895d-b5294aa866fb-host\") pod \"node-ca-8w9d6\" (UID: \"fee8d179-f489-4751-895d-b5294aa866fb\") " pod="openshift-image-registry/node-ca-8w9d6" Aug 01 00:28:33.628922 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.628836 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqz4v\" (UniqueName: \"kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v\") pod \"network-check-target-z87nn\" (UID: \"225ac806-8d39-42b4-91ff-951ed22068f7\") " pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:33.628922 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.628873 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8fb48d9-8b28-4dda-81d3-1401d003e825-host-slash\") pod \"iptables-alerter-jl7q2\" (UID: \"d8fb48d9-8b28-4dda-81d3-1401d003e825\") " pod="openshift-network-operator/iptables-alerter-jl7q2" Aug 01 00:28:33.628922 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.628900 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mhfd\" (UniqueName: \"kubernetes.io/projected/d8fb48d9-8b28-4dda-81d3-1401d003e825-kube-api-access-8mhfd\") pod \"iptables-alerter-jl7q2\" (UID: \"d8fb48d9-8b28-4dda-81d3-1401d003e825\") " pod="openshift-network-operator/iptables-alerter-jl7q2" Aug 01 00:28:33.629098 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.628954 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4e452e0-a47b-4d14-8852-17f8567c98fc-cni-binary-copy\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.629098 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.628993 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4e452e0-a47b-4d14-8852-17f8567c98fc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.629098 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629022 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fee8d179-f489-4751-895d-b5294aa866fb-serviceca\") pod \"node-ca-8w9d6\" (UID: \"fee8d179-f489-4751-895d-b5294aa866fb\") " pod="openshift-image-registry/node-ca-8w9d6" Aug 01 00:28:33.629098 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629047 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a434193d-a324-4f56-90ae-82de780986a4-hosts-file\") pod \"node-resolver-5b74f\" (UID: \"a434193d-a324-4f56-90ae-82de780986a4\") " pod="openshift-dns/node-resolver-5b74f" Aug 01 00:28:33.629098 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629087 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a434193d-a324-4f56-90ae-82de780986a4-tmp-dir\") pod \"node-resolver-5b74f\" (UID: \"a434193d-a324-4f56-90ae-82de780986a4\") " pod="openshift-dns/node-resolver-5b74f" Aug 01 00:28:33.629420 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629136 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4e452e0-a47b-4d14-8852-17f8567c98fc-cnibin\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.629420 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a4e452e0-a47b-4d14-8852-17f8567c98fc-os-release\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.629420 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629183 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9dpq\" (UniqueName: \"kubernetes.io/projected/a4e452e0-a47b-4d14-8852-17f8567c98fc-kube-api-access-b9dpq\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.629420 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629207 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2p27\" (UniqueName: \"kubernetes.io/projected/fee8d179-f489-4751-895d-b5294aa866fb-kube-api-access-p2p27\") pod \"node-ca-8w9d6\" (UID: \"fee8d179-f489-4751-895d-b5294aa866fb\") " pod="openshift-image-registry/node-ca-8w9d6" Aug 01 00:28:33.629420 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629230 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9rbx\" (UniqueName: \"kubernetes.io/projected/8644d164-f68a-4c37-abd5-cc4f66cbe590-kube-api-access-s9rbx\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:33.629420 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629255 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8fb48d9-8b28-4dda-81d3-1401d003e825-iptables-alerter-script\") pod \"iptables-alerter-jl7q2\" (UID: \"d8fb48d9-8b28-4dda-81d3-1401d003e825\") " pod="openshift-network-operator/iptables-alerter-jl7q2" Aug 01 00:28:33.629420 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629281 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4e452e0-a47b-4d14-8852-17f8567c98fc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.629420 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629309 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sfmp\" (UniqueName: \"kubernetes.io/projected/a434193d-a324-4f56-90ae-82de780986a4-kube-api-access-7sfmp\") pod \"node-resolver-5b74f\" (UID: \"a434193d-a324-4f56-90ae-82de780986a4\") " pod="openshift-dns/node-resolver-5b74f" Aug 01 00:28:33.629420 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629333 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:33.629420 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629357 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4e452e0-a47b-4d14-8852-17f8567c98fc-system-cni-dir\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.629420 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.629380 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4e452e0-a47b-4d14-8852-17f8567c98fc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.631038 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.631013 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 01 00:28:33.631038 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.631017 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mz2cv\"" Aug 01 00:28:33.631206 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.631078 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 01 00:28:33.631206 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.631111 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 01 00:28:33.631307 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.631281 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.633689 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.633671 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.633782 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.633734 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 01 00:28:33.633782 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.633748 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r9tnq\"" Aug 01 00:28:33.634039 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.634021 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 01 00:28:33.634110 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.634062 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 01 00:28:33.634187 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.634171 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 01 00:28:33.634299 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.634282 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 01 00:28:33.634571 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.634557 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 01 00:28:33.635977 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.635959 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 01 00:28:33.636038 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.635974 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 01 00:28:33.636210 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.636195 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.636386 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.636364 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 01 00:28:33.636587 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.636567 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fp8hn\"" Aug 01 00:28:33.638522 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.638498 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 01 00:28:33.638522 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.638515 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4bhtw\"" Aug 01 00:28:33.655750 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.655726 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f8cb7" Aug 01 00:28:33.665304 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.665283 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f8cb7" Aug 01 00:28:33.719680 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.719657 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 01 00:28:33.728294 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.728249 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal" event={"ID":"63b803daad13bf5020111e921e186d74","Type":"ContainerStarted","Data":"06a4308425a2c2cdde7204771a909101b72d5b202a5b8268ecb743da0651927d"} Aug 01 00:28:33.729465 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729437 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-195.ec2.internal" event={"ID":"bc8c8050d8f35c84ce46fb25d5f43d30","Type":"ContainerStarted","Data":"7fb4b7d8ab704ad4795e931d2c55dc507f70af04d3cc2a65594e944d14108dfc"} Aug 01 00:28:33.729576 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729553 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-systemd-units\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.729634 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729581 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-run-netns\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.729634 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729608 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7e36f0b-4b54-4945-be93-63ac46767270-ovnkube-config\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.729740 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729632 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-sys\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.729740 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729657 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-host\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.729740 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729681 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/251336a2-faf4-46f3-b301-ea716514b009-konnectivity-ca\") pod \"konnectivity-agent-sbb29\" (UID: \"251336a2-faf4-46f3-b301-ea716514b009\") " pod="kube-system/konnectivity-agent-sbb29" Aug 01 00:28:33.729740 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729705 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-hostroot\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.729740 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729729 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df497cda-451a-4e65-b0a8-b18769d3c9d2-kubelet-config\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:33.730031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729753 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-log-socket\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.730031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729779 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7e36f0b-4b54-4945-be93-63ac46767270-ovnkube-script-lib\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.730031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729811 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9dpq\" (UniqueName: \"kubernetes.io/projected/a4e452e0-a47b-4d14-8852-17f8567c98fc-kube-api-access-b9dpq\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.730031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729852 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqz4v\" (UniqueName: \"kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v\") pod \"network-check-target-z87nn\" (UID: \"225ac806-8d39-42b4-91ff-951ed22068f7\") " pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:33.730031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729876 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-run\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.730031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729899 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-lib-modules\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.730031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729923 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzjw\" (UniqueName: \"kubernetes.io/projected/b09eb022-6d18-49d8-a5ba-51765459875c-kube-api-access-6pzjw\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.730031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729946 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-os-release\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.730031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729972 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-run-multus-certs\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.730031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.729996 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-kubelet\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.730031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730023 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-registration-dir\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.730569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2p27\" (UniqueName: \"kubernetes.io/projected/fee8d179-f489-4751-895d-b5294aa866fb-kube-api-access-p2p27\") pod \"node-ca-8w9d6\" (UID: \"fee8d179-f489-4751-895d-b5294aa866fb\") " pod="openshift-image-registry/node-ca-8w9d6" Aug 01 00:28:33.730569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730085 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rbx\" (UniqueName: \"kubernetes.io/projected/8644d164-f68a-4c37-abd5-cc4f66cbe590-kube-api-access-s9rbx\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:33.730569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730139 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8fb48d9-8b28-4dda-81d3-1401d003e825-iptables-alerter-script\") pod \"iptables-alerter-jl7q2\" (UID: \"d8fb48d9-8b28-4dda-81d3-1401d003e825\") " pod="openshift-network-operator/iptables-alerter-jl7q2" Aug 01 00:28:33.730569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730172 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b09eb022-6d18-49d8-a5ba-51765459875c-cni-binary-copy\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.730569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730198 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-run-netns\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.730569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730282 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-slash\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.730569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730324 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-run-ovn-kubernetes\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.730569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730357 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-socket-dir\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.730569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730392 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sfmp\" (UniqueName: \"kubernetes.io/projected/a434193d-a324-4f56-90ae-82de780986a4-kube-api-access-7sfmp\") pod \"node-resolver-5b74f\" (UID: \"a434193d-a324-4f56-90ae-82de780986a4\") " pod="openshift-dns/node-resolver-5b74f" Aug 01 00:28:33.730569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730422 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-sysconfig\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.730569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730523 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-tuned\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.730569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730550 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-run-k8s-cni-cncf-io\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730588 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df497cda-451a-4e65-b0a8-b18769d3c9d2-dbus\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730621 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-var-lib-openvswitch\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730669 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4e452e0-a47b-4d14-8852-17f8567c98fc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730718 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm2q5\" (UniqueName: \"kubernetes.io/projected/efcb618a-0402-4b4b-95cf-44c48b62abc2-kube-api-access-pm2q5\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730828 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fee8d179-f489-4751-895d-b5294aa866fb-serviceca\") pod \"node-ca-8w9d6\" (UID: \"fee8d179-f489-4751-895d-b5294aa866fb\") " pod="openshift-image-registry/node-ca-8w9d6" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730862 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a434193d-a324-4f56-90ae-82de780986a4-hosts-file\") pod \"node-resolver-5b74f\" (UID: \"a434193d-a324-4f56-90ae-82de780986a4\") " pod="openshift-dns/node-resolver-5b74f" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730889 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4e452e0-a47b-4d14-8852-17f8567c98fc-cnibin\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730915 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a4e452e0-a47b-4d14-8852-17f8567c98fc-os-release\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730945 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-modprobe-d\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730964 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-cni-bin\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730981 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7e36f0b-4b54-4945-be93-63ac46767270-env-overrides\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731002 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7e36f0b-4b54-4945-be93-63ac46767270-ovn-node-metrics-cert\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731025 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a4e452e0-a47b-4d14-8852-17f8567c98fc-os-release\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731033 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4e452e0-a47b-4d14-8852-17f8567c98fc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731060 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-kubernetes\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.731103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731098 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-system-cni-dir\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731099 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4e452e0-a47b-4d14-8852-17f8567c98fc-cnibin\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731133 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-cnibin\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.730961 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a434193d-a324-4f56-90ae-82de780986a4-hosts-file\") pod \"node-resolver-5b74f\" (UID: \"a434193d-a324-4f56-90ae-82de780986a4\") " pod="openshift-dns/node-resolver-5b74f" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731230 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-var-lib-cni-bin\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731258 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-var-lib-cni-multus\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731283 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731307 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-etc-openvswitch\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731324 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4e452e0-a47b-4d14-8852-17f8567c98fc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731334 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731375 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fee8d179-f489-4751-895d-b5294aa866fb-serviceca\") pod \"node-ca-8w9d6\" (UID: \"fee8d179-f489-4751-895d-b5294aa866fb\") " pod="openshift-image-registry/node-ca-8w9d6" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731376 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4e452e0-a47b-4d14-8852-17f8567c98fc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:33.731421 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731431 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-run-systemd\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731458 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-cni-netd\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:33.731488 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs podName:8644d164-f68a-4c37-abd5-cc4f66cbe590 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:34.231465744 +0000 UTC m=+3.020187562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs") pod "network-metrics-daemon-wkj56" (UID: "8644d164-f68a-4c37-abd5-cc4f66cbe590") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:28:33.731913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731519 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731544 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731548 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4e452e0-a47b-4d14-8852-17f8567c98fc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731570 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fee8d179-f489-4751-895d-b5294aa866fb-host\") pod \"node-ca-8w9d6\" (UID: \"fee8d179-f489-4751-895d-b5294aa866fb\") " pod="openshift-image-registry/node-ca-8w9d6" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731604 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8fb48d9-8b28-4dda-81d3-1401d003e825-host-slash\") pod \"iptables-alerter-jl7q2\" (UID: \"d8fb48d9-8b28-4dda-81d3-1401d003e825\") " pod="openshift-network-operator/iptables-alerter-jl7q2" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731630 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mhfd\" (UniqueName: \"kubernetes.io/projected/d8fb48d9-8b28-4dda-81d3-1401d003e825-kube-api-access-8mhfd\") pod \"iptables-alerter-jl7q2\" (UID: \"d8fb48d9-8b28-4dda-81d3-1401d003e825\") " pod="openshift-network-operator/iptables-alerter-jl7q2" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731656 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4e452e0-a47b-4d14-8852-17f8567c98fc-cni-binary-copy\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731662 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4e452e0-a47b-4d14-8852-17f8567c98fc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731679 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-var-lib-kubelet\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731690 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8fb48d9-8b28-4dda-81d3-1401d003e825-host-slash\") pod \"iptables-alerter-jl7q2\" (UID: \"d8fb48d9-8b28-4dda-81d3-1401d003e825\") " pod="openshift-network-operator/iptables-alerter-jl7q2" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731705 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbgw8\" (UniqueName: \"kubernetes.io/projected/cd7a3121-0e51-4f13-8d05-c0b10f115420-kube-api-access-sbgw8\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731717 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fee8d179-f489-4751-895d-b5294aa866fb-host\") pod \"node-ca-8w9d6\" (UID: \"fee8d179-f489-4751-895d-b5294aa866fb\") " pod="openshift-image-registry/node-ca-8w9d6" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731724 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8fb48d9-8b28-4dda-81d3-1401d003e825-iptables-alerter-script\") pod \"iptables-alerter-jl7q2\" (UID: \"d8fb48d9-8b28-4dda-81d3-1401d003e825\") " pod="openshift-network-operator/iptables-alerter-jl7q2" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731742 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/251336a2-faf4-46f3-b301-ea716514b009-agent-certs\") pod \"konnectivity-agent-sbb29\" (UID: \"251336a2-faf4-46f3-b301-ea716514b009\") " pod="kube-system/konnectivity-agent-sbb29" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731789 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-run-openvswitch\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731822 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a434193d-a324-4f56-90ae-82de780986a4-tmp-dir\") pod \"node-resolver-5b74f\" (UID: \"a434193d-a324-4f56-90ae-82de780986a4\") " pod="openshift-dns/node-resolver-5b74f" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731847 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-var-lib-kubelet\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.732593 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731952 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-etc-kubernetes\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.731986 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-run-ovn\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732033 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-sysctl-conf\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732058 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b09eb022-6d18-49d8-a5ba-51765459875c-multus-daemon-config\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732118 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4e452e0-a47b-4d14-8852-17f8567c98fc-system-cni-dir\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732144 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-sysctl-d\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732146 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a434193d-a324-4f56-90ae-82de780986a4-tmp-dir\") pod \"node-resolver-5b74f\" (UID: \"a434193d-a324-4f56-90ae-82de780986a4\") " pod="openshift-dns/node-resolver-5b74f" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732173 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxmwh\" (UniqueName: \"kubernetes.io/projected/e7e36f0b-4b54-4945-be93-63ac46767270-kube-api-access-mxmwh\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732200 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4e452e0-a47b-4d14-8852-17f8567c98fc-cni-binary-copy\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732225 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4e452e0-a47b-4d14-8852-17f8567c98fc-system-cni-dir\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732220 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-sys-fs\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732268 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd7a3121-0e51-4f13-8d05-c0b10f115420-tmp\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732297 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-multus-socket-dir-parent\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732323 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-multus-conf-dir\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732350 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-multus-cni-dir\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732374 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-node-log\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732402 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.733202 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732432 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-device-dir\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.733845 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.732458 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-systemd\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.738679 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:33.738659 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:28:33.738832 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:33.738684 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:28:33.738832 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:33.738695 2563 projected.go:196] Error preparing data for projected volume kube-api-access-kqz4v for pod openshift-network-diagnostics/network-check-target-z87nn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:28:33.738832 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:33.738748 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v podName:225ac806-8d39-42b4-91ff-951ed22068f7 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:34.238735782 +0000 UTC m=+3.027457591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kqz4v" (UniqueName: "kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v") pod "network-check-target-z87nn" (UID: "225ac806-8d39-42b4-91ff-951ed22068f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:28:33.739004 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.738858 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 01 00:28:33.743373 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.743286 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sfmp\" (UniqueName: \"kubernetes.io/projected/a434193d-a324-4f56-90ae-82de780986a4-kube-api-access-7sfmp\") pod \"node-resolver-5b74f\" (UID: \"a434193d-a324-4f56-90ae-82de780986a4\") " pod="openshift-dns/node-resolver-5b74f" Aug 01 00:28:33.743373 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.743329 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mhfd\" (UniqueName: \"kubernetes.io/projected/d8fb48d9-8b28-4dda-81d3-1401d003e825-kube-api-access-8mhfd\") pod \"iptables-alerter-jl7q2\" (UID: \"d8fb48d9-8b28-4dda-81d3-1401d003e825\") " pod="openshift-network-operator/iptables-alerter-jl7q2" Aug 01 00:28:33.743373 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.743286 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rbx\" (UniqueName: \"kubernetes.io/projected/8644d164-f68a-4c37-abd5-cc4f66cbe590-kube-api-access-s9rbx\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:33.743373 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.743291 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9dpq\" (UniqueName: \"kubernetes.io/projected/a4e452e0-a47b-4d14-8852-17f8567c98fc-kube-api-access-b9dpq\") pod \"multus-additional-cni-plugins-q2wgc\" (UID: \"a4e452e0-a47b-4d14-8852-17f8567c98fc\") " pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.743568 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.743381 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2p27\" (UniqueName: \"kubernetes.io/projected/fee8d179-f489-4751-895d-b5294aa866fb-kube-api-access-p2p27\") pod \"node-ca-8w9d6\" (UID: \"fee8d179-f489-4751-895d-b5294aa866fb\") " pod="openshift-image-registry/node-ca-8w9d6" Aug 01 00:28:33.797247 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.797216 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:28:33.832794 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.832760 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-sys\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.832971 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.832801 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-host\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.832971 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.832828 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/251336a2-faf4-46f3-b301-ea716514b009-konnectivity-ca\") pod \"konnectivity-agent-sbb29\" (UID: \"251336a2-faf4-46f3-b301-ea716514b009\") " pod="kube-system/konnectivity-agent-sbb29" Aug 01 00:28:33.832971 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.832852 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-hostroot\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.832971 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.832876 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df497cda-451a-4e65-b0a8-b18769d3c9d2-kubelet-config\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:33.832971 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.832902 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-log-socket\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.832971 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.832927 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7e36f0b-4b54-4945-be93-63ac46767270-ovnkube-script-lib\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.832971 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.832936 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-host\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.832971 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.832970 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-run\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833013 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-lib-modules\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833014 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df497cda-451a-4e65-b0a8-b18769d3c9d2-kubelet-config\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833042 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzjw\" (UniqueName: \"kubernetes.io/projected/b09eb022-6d18-49d8-a5ba-51765459875c-kube-api-access-6pzjw\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833076 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-hostroot\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833097 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-os-release\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833125 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-run-multus-certs\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833169 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-log-socket\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833173 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-kubelet\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833209 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-lib-modules\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833220 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-kubelet\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-registration-dir\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833265 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-sys\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833317 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-registration-dir\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833366 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-os-release\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.833391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833127 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-run\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833412 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-run-multus-certs\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833436 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b09eb022-6d18-49d8-a5ba-51765459875c-cni-binary-copy\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833468 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-run-netns\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833486 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/251336a2-faf4-46f3-b301-ea716514b009-konnectivity-ca\") pod \"konnectivity-agent-sbb29\" (UID: \"251336a2-faf4-46f3-b301-ea716514b009\") " pod="kube-system/konnectivity-agent-sbb29" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-slash\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833522 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-run-ovn-kubernetes\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833550 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-socket-dir\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833556 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-slash\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833562 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-run-netns\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833577 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-sysconfig\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833603 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-tuned\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833627 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-run-k8s-cni-cncf-io\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833637 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-sysconfig\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833653 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df497cda-451a-4e65-b0a8-b18769d3c9d2-dbus\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833694 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-var-lib-openvswitch\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833716 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7e36f0b-4b54-4945-be93-63ac46767270-ovnkube-script-lib\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.834171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833725 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2q5\" (UniqueName: \"kubernetes.io/projected/efcb618a-0402-4b4b-95cf-44c48b62abc2-kube-api-access-pm2q5\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833766 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-var-lib-openvswitch\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833765 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-modprobe-d\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833783 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df497cda-451a-4e65-b0a8-b18769d3c9d2-dbus\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833793 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-cni-bin\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833823 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-cni-bin\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833834 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-run-k8s-cni-cncf-io\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833846 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-modprobe-d\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833866 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7e36f0b-4b54-4945-be93-63ac46767270-env-overrides\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833891 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7e36f0b-4b54-4945-be93-63ac46767270-ovn-node-metrics-cert\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833898 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-run-ovn-kubernetes\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833926 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-kubernetes\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833955 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-system-cni-dir\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.833980 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-cnibin\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834002 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b09eb022-6d18-49d8-a5ba-51765459875c-cni-binary-copy\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834004 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-var-lib-cni-bin\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834042 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-var-lib-cni-multus\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834061 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-system-cni-dir\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.835149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834069 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834097 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-etc-openvswitch\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834108 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-cnibin\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834117 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-var-lib-cni-multus\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834156 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-etc-openvswitch\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834161 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-run-systemd\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834057 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-kubernetes\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834189 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-cni-netd\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:33.834203 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834209 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-var-lib-cni-bin\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834215 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834240 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834248 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-run-systemd\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:33.834264 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret podName:df497cda-451a-4e65-b0a8-b18769d3c9d2 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:34.334246999 +0000 UTC m=+3.122968820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret") pod "global-pull-secret-syncer-nlw9l" (UID: "df497cda-451a-4e65-b0a8-b18769d3c9d2") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-var-lib-kubelet\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834324 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbgw8\" (UniqueName: \"kubernetes.io/projected/cd7a3121-0e51-4f13-8d05-c0b10f115420-kube-api-access-sbgw8\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834327 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.836130 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834348 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/251336a2-faf4-46f3-b301-ea716514b009-agent-certs\") pod \"konnectivity-agent-sbb29\" (UID: \"251336a2-faf4-46f3-b301-ea716514b009\") " pod="kube-system/konnectivity-agent-sbb29" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834369 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-cni-netd\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834372 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7e36f0b-4b54-4945-be93-63ac46767270-env-overrides\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834376 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-run-openvswitch\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834426 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-var-lib-kubelet\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834429 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-var-lib-kubelet\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834443 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834478 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-host-var-lib-kubelet\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834488 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-run-openvswitch\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834510 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-socket-dir\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834519 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-etc-kubernetes\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834576 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-run-ovn\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834603 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-sysctl-conf\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834606 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-etc-kubernetes\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834652 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-run-ovn\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834732 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b09eb022-6d18-49d8-a5ba-51765459875c-multus-daemon-config\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834754 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-sysctl-conf\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834765 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-sysctl-d\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.837012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834793 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxmwh\" (UniqueName: \"kubernetes.io/projected/e7e36f0b-4b54-4945-be93-63ac46767270-kube-api-access-mxmwh\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834817 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-sys-fs\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834840 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd7a3121-0e51-4f13-8d05-c0b10f115420-tmp\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834866 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-multus-socket-dir-parent\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834892 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-multus-conf-dir\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834910 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-sysctl-d\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834926 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-multus-cni-dir\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834952 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-node-log\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.834978 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835004 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-device-dir\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835027 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-systemd\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835049 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-systemd-units\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-run-netns\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835114 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7e36f0b-4b54-4945-be93-63ac46767270-ovnkube-config\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835253 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b09eb022-6d18-49d8-a5ba-51765459875c-multus-daemon-config\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835331 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835392 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-multus-socket-dir-parent\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835448 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-multus-conf-dir\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.838764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835602 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-node-log\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.838764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835625 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7e36f0b-4b54-4945-be93-63ac46767270-ovnkube-config\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.838764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835625 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b09eb022-6d18-49d8-a5ba-51765459875c-multus-cni-dir\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.838764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835680 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-sys-fs\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.838764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835706 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/efcb618a-0402-4b4b-95cf-44c48b62abc2-device-dir\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.838764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835727 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-systemd-units\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.838764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835743 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7e36f0b-4b54-4945-be93-63ac46767270-host-run-netns\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.838764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.835782 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-systemd\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.838764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.837808 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/251336a2-faf4-46f3-b301-ea716514b009-agent-certs\") pod \"konnectivity-agent-sbb29\" (UID: \"251336a2-faf4-46f3-b301-ea716514b009\") " pod="kube-system/konnectivity-agent-sbb29" Aug 01 00:28:33.838764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.838438 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7e36f0b-4b54-4945-be93-63ac46767270-ovn-node-metrics-cert\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.839296 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.838813 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd7a3121-0e51-4f13-8d05-c0b10f115420-tmp\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.839296 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.838966 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd7a3121-0e51-4f13-8d05-c0b10f115420-etc-tuned\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.844076 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.844049 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbgw8\" (UniqueName: \"kubernetes.io/projected/cd7a3121-0e51-4f13-8d05-c0b10f115420-kube-api-access-sbgw8\") pod \"tuned-69l6n\" (UID: \"cd7a3121-0e51-4f13-8d05-c0b10f115420\") " pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.844637 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.844613 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxmwh\" (UniqueName: \"kubernetes.io/projected/e7e36f0b-4b54-4945-be93-63ac46767270-kube-api-access-mxmwh\") pod \"ovnkube-node-b54j8\" (UID: \"e7e36f0b-4b54-4945-be93-63ac46767270\") " pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.845392 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.845338 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzjw\" (UniqueName: \"kubernetes.io/projected/b09eb022-6d18-49d8-a5ba-51765459875c-kube-api-access-6pzjw\") pod \"multus-gx8gw\" (UID: \"b09eb022-6d18-49d8-a5ba-51765459875c\") " pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.847062 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.847037 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2q5\" (UniqueName: \"kubernetes.io/projected/efcb618a-0402-4b4b-95cf-44c48b62abc2-kube-api-access-pm2q5\") pod \"aws-ebs-csi-driver-node-xzcj7\" (UID: \"efcb618a-0402-4b4b-95cf-44c48b62abc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.918468 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.918430 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8w9d6" Aug 01 00:28:33.927575 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.927551 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5b74f" Aug 01 00:28:33.933094 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:33.933063 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee8d179_f489_4751_895d_b5294aa866fb.slice/crio-5c6ecaa08dc1619699d80e743606013fa755cc0cfd384f3e203aa836f4a3ded8 WatchSource:0}: Error finding container 5c6ecaa08dc1619699d80e743606013fa755cc0cfd384f3e203aa836f4a3ded8: Status 404 returned error can't find the container with id 5c6ecaa08dc1619699d80e743606013fa755cc0cfd384f3e203aa836f4a3ded8 Aug 01 00:28:33.936007 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.935634 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jl7q2" Aug 01 00:28:33.939558 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:33.939532 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda434193d_a324_4f56_90ae_82de780986a4.slice/crio-55cb111f914f4d202974fef148efec16592889f9236b74c8594bd66ba6272f5a WatchSource:0}: Error finding container 55cb111f914f4d202974fef148efec16592889f9236b74c8594bd66ba6272f5a: Status 404 returned error can't find the container with id 55cb111f914f4d202974fef148efec16592889f9236b74c8594bd66ba6272f5a Aug 01 00:28:33.940569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.940547 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q2wgc" Aug 01 00:28:33.948112 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:33.947823 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fb48d9_8b28_4dda_81d3_1401d003e825.slice/crio-dd7306d4aca3517ddd51ed12e987cec988d4fe2a43fa9e7d93aa16987e68129a WatchSource:0}: Error finding container dd7306d4aca3517ddd51ed12e987cec988d4fe2a43fa9e7d93aa16987e68129a: Status 404 returned error can't find the container with id dd7306d4aca3517ddd51ed12e987cec988d4fe2a43fa9e7d93aa16987e68129a Aug 01 00:28:33.948112 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.947953 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-69l6n" Aug 01 00:28:33.952087 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:33.952064 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e452e0_a47b_4d14_8852_17f8567c98fc.slice/crio-5cbf84482c87091655e89873004a214871ae5bdbee32e53cfe2b6185f6f739d9 WatchSource:0}: Error finding container 5cbf84482c87091655e89873004a214871ae5bdbee32e53cfe2b6185f6f739d9: Status 404 returned error can't find the container with id 5cbf84482c87091655e89873004a214871ae5bdbee32e53cfe2b6185f6f739d9 Aug 01 00:28:33.954598 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.954568 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sbb29" Aug 01 00:28:33.960119 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:33.960094 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7a3121_0e51_4f13_8d05_c0b10f115420.slice/crio-a6487e53ab7b8fddb15ceaf6ba50244ceccfa564f51b454e3d4c36ff76a7744a WatchSource:0}: Error finding container a6487e53ab7b8fddb15ceaf6ba50244ceccfa564f51b454e3d4c36ff76a7744a: Status 404 returned error can't find the container with id a6487e53ab7b8fddb15ceaf6ba50244ceccfa564f51b454e3d4c36ff76a7744a Aug 01 00:28:33.962797 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.962776 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:33.965275 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:33.965254 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251336a2_faf4_46f3_b301_ea716514b009.slice/crio-608fc4ad6a09b314fee28bd4f133202080ebe3447a48b3dc96062924d45604db WatchSource:0}: Error finding container 608fc4ad6a09b314fee28bd4f133202080ebe3447a48b3dc96062924d45604db: Status 404 returned error can't find the container with id 608fc4ad6a09b314fee28bd4f133202080ebe3447a48b3dc96062924d45604db Aug 01 00:28:33.969720 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.969698 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" Aug 01 00:28:33.975429 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:33.975384 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gx8gw" Aug 01 00:28:33.980246 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:33.980204 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e36f0b_4b54_4945_be93_63ac46767270.slice/crio-49067d976f21d12e337b6fc1fa8deac8e156c816f39fa2968acec3d046724f20 WatchSource:0}: Error finding container 49067d976f21d12e337b6fc1fa8deac8e156c816f39fa2968acec3d046724f20: Status 404 returned error can't find the container with id 49067d976f21d12e337b6fc1fa8deac8e156c816f39fa2968acec3d046724f20 Aug 01 00:28:33.980595 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:33.980570 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefcb618a_0402_4b4b_95cf_44c48b62abc2.slice/crio-e12f16fdf0b432f332376509cd0d82d26897352d11abf5e7b7ea0782be2add8f WatchSource:0}: Error finding container e12f16fdf0b432f332376509cd0d82d26897352d11abf5e7b7ea0782be2add8f: Status 404 returned error can't find the container with id e12f16fdf0b432f332376509cd0d82d26897352d11abf5e7b7ea0782be2add8f Aug 01 00:28:33.987519 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:28:33.987457 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09eb022_6d18_49d8_a5ba_51765459875c.slice/crio-49bdfe3493a97eb8658edde997759f288396b8d2500798b765012b2839838d7d WatchSource:0}: Error finding container 49bdfe3493a97eb8658edde997759f288396b8d2500798b765012b2839838d7d: Status 404 returned error can't find the container with id 49bdfe3493a97eb8658edde997759f288396b8d2500798b765012b2839838d7d Aug 01 00:28:34.239113 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.239016 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqz4v\" (UniqueName: \"kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v\") pod \"network-check-target-z87nn\" (UID: \"225ac806-8d39-42b4-91ff-951ed22068f7\") " pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:34.239113 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.239100 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:34.239338 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:34.239224 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:28:34.239338 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:34.239247 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:28:34.239338 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:34.239273 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:28:34.239338 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:34.239288 2563 projected.go:196] Error preparing data for projected volume kube-api-access-kqz4v for pod openshift-network-diagnostics/network-check-target-z87nn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:28:34.239338 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:34.239295 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs podName:8644d164-f68a-4c37-abd5-cc4f66cbe590 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:35.23927461 +0000 UTC m=+4.027996436 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs") pod "network-metrics-daemon-wkj56" (UID: "8644d164-f68a-4c37-abd5-cc4f66cbe590") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:28:34.239531 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:34.239358 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v podName:225ac806-8d39-42b4-91ff-951ed22068f7 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:35.239340621 +0000 UTC m=+4.028062428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kqz4v" (UniqueName: "kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v") pod "network-check-target-z87nn" (UID: "225ac806-8d39-42b4-91ff-951ed22068f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:28:34.340060 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.340027 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:34.340228 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:34.340182 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:28:34.340304 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:34.340247 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret podName:df497cda-451a-4e65-b0a8-b18769d3c9d2 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:35.340228836 +0000 UTC m=+4.128950641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret") pod "global-pull-secret-syncer-nlw9l" (UID: "df497cda-451a-4e65-b0a8-b18769d3c9d2") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:28:34.670015 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.669894 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-31 00:23:34 +0000 UTC" deadline="2028-01-06 01:19:40.802563192 +0000 UTC" Aug 01 00:28:34.670015 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.669962 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12552h51m6.132606345s" Aug 01 00:28:34.722487 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.722456 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:34.722673 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:34.722603 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:34.733004 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.732966 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" event={"ID":"efcb618a-0402-4b4b-95cf-44c48b62abc2","Type":"ContainerStarted","Data":"e12f16fdf0b432f332376509cd0d82d26897352d11abf5e7b7ea0782be2add8f"} Aug 01 00:28:34.735861 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.735759 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" event={"ID":"e7e36f0b-4b54-4945-be93-63ac46767270","Type":"ContainerStarted","Data":"49067d976f21d12e337b6fc1fa8deac8e156c816f39fa2968acec3d046724f20"} Aug 01 00:28:34.739252 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.739228 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sbb29" event={"ID":"251336a2-faf4-46f3-b301-ea716514b009","Type":"ContainerStarted","Data":"608fc4ad6a09b314fee28bd4f133202080ebe3447a48b3dc96062924d45604db"} Aug 01 00:28:34.741112 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.741086 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q2wgc" event={"ID":"a4e452e0-a47b-4d14-8852-17f8567c98fc","Type":"ContainerStarted","Data":"5cbf84482c87091655e89873004a214871ae5bdbee32e53cfe2b6185f6f739d9"} Aug 01 00:28:34.743408 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.743364 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5b74f" event={"ID":"a434193d-a324-4f56-90ae-82de780986a4","Type":"ContainerStarted","Data":"55cb111f914f4d202974fef148efec16592889f9236b74c8594bd66ba6272f5a"} Aug 01 00:28:34.745212 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.745163 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gx8gw" event={"ID":"b09eb022-6d18-49d8-a5ba-51765459875c","Type":"ContainerStarted","Data":"49bdfe3493a97eb8658edde997759f288396b8d2500798b765012b2839838d7d"} Aug 01 00:28:34.747311 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.747286 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-69l6n" event={"ID":"cd7a3121-0e51-4f13-8d05-c0b10f115420","Type":"ContainerStarted","Data":"a6487e53ab7b8fddb15ceaf6ba50244ceccfa564f51b454e3d4c36ff76a7744a"} Aug 01 00:28:34.749686 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.749661 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jl7q2" event={"ID":"d8fb48d9-8b28-4dda-81d3-1401d003e825","Type":"ContainerStarted","Data":"dd7306d4aca3517ddd51ed12e987cec988d4fe2a43fa9e7d93aa16987e68129a"} Aug 01 00:28:34.751478 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:34.751455 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8w9d6" event={"ID":"fee8d179-f489-4751-895d-b5294aa866fb","Type":"ContainerStarted","Data":"5c6ecaa08dc1619699d80e743606013fa755cc0cfd384f3e203aa836f4a3ded8"} Aug 01 00:28:35.248168 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:35.248128 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqz4v\" (UniqueName: \"kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v\") pod \"network-check-target-z87nn\" (UID: \"225ac806-8d39-42b4-91ff-951ed22068f7\") " pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:35.248349 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:35.248217 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:35.248439 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:35.248361 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:28:35.248514 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:35.248463 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs podName:8644d164-f68a-4c37-abd5-cc4f66cbe590 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:37.248442719 +0000 UTC m=+6.037164540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs") pod "network-metrics-daemon-wkj56" (UID: "8644d164-f68a-4c37-abd5-cc4f66cbe590") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:28:35.248917 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:35.248890 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:28:35.248917 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:35.248928 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:28:35.248917 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:35.248941 2563 projected.go:196] Error preparing data for projected volume kube-api-access-kqz4v for pod openshift-network-diagnostics/network-check-target-z87nn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:28:35.249280 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:35.248989 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v podName:225ac806-8d39-42b4-91ff-951ed22068f7 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:37.248972364 +0000 UTC m=+6.037694182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kqz4v" (UniqueName: "kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v") pod "network-check-target-z87nn" (UID: "225ac806-8d39-42b4-91ff-951ed22068f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:28:35.349586 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:35.349546 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:35.349754 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:35.349708 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:28:35.349831 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:35.349778 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret podName:df497cda-451a-4e65-b0a8-b18769d3c9d2 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:37.349759612 +0000 UTC m=+6.138481437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret") pod "global-pull-secret-syncer-nlw9l" (UID: "df497cda-451a-4e65-b0a8-b18769d3c9d2") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:28:35.670651 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:35.670501 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-31 00:23:34 +0000 UTC" deadline="2028-04-05 08:41:39.474378134 +0000 UTC" Aug 01 00:28:35.670651 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:35.670544 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14720h13m3.803839073s" Aug 01 00:28:35.724862 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:35.724819 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:35.725022 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:35.724958 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:35.725418 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:35.725398 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:35.725520 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:35.725498 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:36.722443 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:36.722405 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:36.723064 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:36.722570 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:36.773123 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:36.773083 2563 generic.go:356] "Generic (PLEG): container finished" podID="63b803daad13bf5020111e921e186d74" containerID="3d3cfcf4ca97d6e0be94fa44098aef2da9c7f22a9f66770730ebf208aa3ae691" exitCode=0 Aug 01 00:28:36.773316 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:36.773210 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal" event={"ID":"63b803daad13bf5020111e921e186d74","Type":"ContainerDied","Data":"3d3cfcf4ca97d6e0be94fa44098aef2da9c7f22a9f66770730ebf208aa3ae691"} Aug 01 00:28:36.793874 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:36.793833 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-195.ec2.internal" event={"ID":"bc8c8050d8f35c84ce46fb25d5f43d30","Type":"ContainerStarted","Data":"94dceddd528e2d6a16d8c04f6535150828f3aca8504548e2e4b0dd78215de6f2"} Aug 01 00:28:36.808152 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:36.808065 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-195.ec2.internal" podStartSLOduration=2.8080447 podStartE2EDuration="2.8080447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:28:36.807437029 +0000 UTC m=+5.596158859" watchObservedRunningTime="2026-08-01 00:28:36.8080447 +0000 UTC m=+5.596766530" Aug 01 00:28:37.267410 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:37.267327 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:37.267569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:37.267415 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqz4v\" (UniqueName: \"kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v\") pod \"network-check-target-z87nn\" (UID: \"225ac806-8d39-42b4-91ff-951ed22068f7\") " pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:37.267631 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:37.267583 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:28:37.267631 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:37.267608 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:28:37.267631 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:37.267629 2563 projected.go:196] Error preparing data for projected volume kube-api-access-kqz4v for pod openshift-network-diagnostics/network-check-target-z87nn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:28:37.267821 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:37.267710 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v podName:225ac806-8d39-42b4-91ff-951ed22068f7 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:41.267683486 +0000 UTC m=+10.056405312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kqz4v" (UniqueName: "kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v") pod "network-check-target-z87nn" (UID: "225ac806-8d39-42b4-91ff-951ed22068f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:28:37.267821 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:37.267814 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:28:37.267941 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:37.267857 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs podName:8644d164-f68a-4c37-abd5-cc4f66cbe590 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:41.2678454 +0000 UTC m=+10.056567221 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs") pod "network-metrics-daemon-wkj56" (UID: "8644d164-f68a-4c37-abd5-cc4f66cbe590") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:28:37.368782 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:37.368112 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:37.368782 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:37.368332 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:28:37.368782 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:37.368395 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret podName:df497cda-451a-4e65-b0a8-b18769d3c9d2 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:41.368374402 +0000 UTC m=+10.157096247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret") pod "global-pull-secret-syncer-nlw9l" (UID: "df497cda-451a-4e65-b0a8-b18769d3c9d2") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:28:37.723832 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:37.723235 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:37.723832 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:37.723375 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:37.724378 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:37.723894 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:37.724378 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:37.724025 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:38.722080 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:38.722033 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:38.722262 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:38.722213 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:39.722919 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:39.722874 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:39.723383 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:39.722881 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:39.723383 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:39.722993 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:39.723383 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:39.723116 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:40.722246 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:40.722207 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:40.722434 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:40.722352 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:41.304519 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:41.304477 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqz4v\" (UniqueName: \"kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v\") pod \"network-check-target-z87nn\" (UID: \"225ac806-8d39-42b4-91ff-951ed22068f7\") " pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:41.305005 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:41.304568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:41.305005 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:41.304701 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:28:41.305005 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:41.304766 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs podName:8644d164-f68a-4c37-abd5-cc4f66cbe590 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:49.304744437 +0000 UTC m=+18.093466259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs") pod "network-metrics-daemon-wkj56" (UID: "8644d164-f68a-4c37-abd5-cc4f66cbe590") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:28:41.305279 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:41.305225 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:28:41.305279 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:41.305247 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:28:41.305279 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:41.305263 2563 projected.go:196] Error preparing data for projected volume kube-api-access-kqz4v for pod openshift-network-diagnostics/network-check-target-z87nn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:28:41.305435 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:41.305317 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v podName:225ac806-8d39-42b4-91ff-951ed22068f7 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:49.305301029 +0000 UTC m=+18.094022838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kqz4v" (UniqueName: "kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v") pod "network-check-target-z87nn" (UID: "225ac806-8d39-42b4-91ff-951ed22068f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:28:41.405384 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:41.405347 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:41.405580 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:41.405523 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:28:41.405646 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:41.405603 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret podName:df497cda-451a-4e65-b0a8-b18769d3c9d2 nodeName:}" failed. No retries permitted until 2026-08-01 00:28:49.405583015 +0000 UTC m=+18.194304837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret") pod "global-pull-secret-syncer-nlw9l" (UID: "df497cda-451a-4e65-b0a8-b18769d3c9d2") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:28:41.724261 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:41.723552 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:41.724261 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:41.723667 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:41.724261 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:41.724054 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:41.724261 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:41.724168 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:42.722818 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:42.722783 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:42.723434 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:42.722936 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:43.723147 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:43.722626 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:43.723147 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:43.722637 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:43.723147 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:43.722769 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:43.723147 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:43.722882 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:44.721987 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:44.721952 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:44.722187 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:44.722081 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:45.722644 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:45.722608 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:45.723088 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:45.722608 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:45.723088 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:45.722742 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:45.723088 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:45.722788 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:46.721934 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:46.721904 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:46.722114 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:46.722079 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:47.722836 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:47.722772 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:47.723312 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:47.722923 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:47.723312 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:47.722977 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:47.723312 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:47.723095 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:48.722633 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:48.722603 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:48.722870 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:48.722728 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:49.369560 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:49.369529 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:49.369710 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:49.369571 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqz4v\" (UniqueName: \"kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v\") pod \"network-check-target-z87nn\" (UID: \"225ac806-8d39-42b4-91ff-951ed22068f7\") " pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:49.369710 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:49.369657 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:28:49.369710 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:49.369673 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:28:49.369710 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:49.369685 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:28:49.369710 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:49.369693 2563 projected.go:196] Error preparing data for projected volume kube-api-access-kqz4v for pod openshift-network-diagnostics/network-check-target-z87nn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:28:49.369876 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:49.369717 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs podName:8644d164-f68a-4c37-abd5-cc4f66cbe590 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:05.369701188 +0000 UTC m=+34.158422990 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs") pod "network-metrics-daemon-wkj56" (UID: "8644d164-f68a-4c37-abd5-cc4f66cbe590") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:28:49.369876 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:49.369731 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v podName:225ac806-8d39-42b4-91ff-951ed22068f7 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:05.369724775 +0000 UTC m=+34.158446578 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kqz4v" (UniqueName: "kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v") pod "network-check-target-z87nn" (UID: "225ac806-8d39-42b4-91ff-951ed22068f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:28:49.469992 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:49.469962 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:49.470131 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:49.470114 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:28:49.470175 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:49.470170 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret podName:df497cda-451a-4e65-b0a8-b18769d3c9d2 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:05.470157028 +0000 UTC m=+34.258878831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret") pod "global-pull-secret-syncer-nlw9l" (UID: "df497cda-451a-4e65-b0a8-b18769d3c9d2") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:28:49.722289 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:49.722209 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:49.722445 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:49.722215 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:49.722445 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:49.722339 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:49.722445 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:49.722415 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:50.722637 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:50.722607 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:50.723043 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:50.722746 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:51.723199 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:51.723166 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:51.723634 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:51.723254 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:51.723634 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:51.723281 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:51.723634 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:51.723366 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:52.722466 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:52.722432 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:52.722628 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:52.722559 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:53.722283 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.722132 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:53.722833 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.722189 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:53.722833 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:53.722360 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:53.722833 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:53.722424 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:53.827225 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.827074 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" event={"ID":"efcb618a-0402-4b4b-95cf-44c48b62abc2","Type":"ContainerStarted","Data":"4a0b74249ca552d4528a6bcde6d60c99296757dddc2c68b10f1975ca602c5b7b"} Aug 01 00:28:53.828741 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.828723 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:28:53.829030 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.829012 2563 generic.go:356] "Generic (PLEG): container finished" podID="e7e36f0b-4b54-4945-be93-63ac46767270" containerID="1f5d8924458db543e751c4837adb9b6acd4ba4e8a441294ab16129d865f3b5bd" exitCode=1 Aug 01 00:28:53.829108 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.829084 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" event={"ID":"e7e36f0b-4b54-4945-be93-63ac46767270","Type":"ContainerStarted","Data":"18e14bcc4daa4246cfab19bfa06d9c9888c30619299dfe2793eaefa5bc7425d4"} Aug 01 00:28:53.829174 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.829159 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" event={"ID":"e7e36f0b-4b54-4945-be93-63ac46767270","Type":"ContainerDied","Data":"1f5d8924458db543e751c4837adb9b6acd4ba4e8a441294ab16129d865f3b5bd"} Aug 01 00:28:53.829230 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.829217 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" event={"ID":"e7e36f0b-4b54-4945-be93-63ac46767270","Type":"ContainerStarted","Data":"7f7bfe751036619398a300532db5f782545e204b1a07b041e0cf855bd8f0fc28"} Aug 01 00:28:53.830471 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.830368 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sbb29" event={"ID":"251336a2-faf4-46f3-b301-ea716514b009","Type":"ContainerStarted","Data":"e70e667faf73708564eda0939f317ee0779cf7082ea781881a185ed09b4d64da"} Aug 01 00:28:53.831944 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.831918 2563 generic.go:356] "Generic (PLEG): container finished" podID="a4e452e0-a47b-4d14-8852-17f8567c98fc" containerID="04fe32c887e8ff8d132cdfadd20d135b4b3a8ec97b9376ce0bcd72452040ae14" exitCode=0 Aug 01 00:28:53.832070 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.831946 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q2wgc" event={"ID":"a4e452e0-a47b-4d14-8852-17f8567c98fc","Type":"ContainerDied","Data":"04fe32c887e8ff8d132cdfadd20d135b4b3a8ec97b9376ce0bcd72452040ae14"} Aug 01 00:28:53.833442 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.833422 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5b74f" event={"ID":"a434193d-a324-4f56-90ae-82de780986a4","Type":"ContainerStarted","Data":"8b45f4f6d09280bff14480b3c4af3725f4a3e3a143d85507ca23d013a423e2b7"} Aug 01 00:28:53.835558 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.835436 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal" event={"ID":"63b803daad13bf5020111e921e186d74","Type":"ContainerStarted","Data":"67e50eb5aaf7fecc12ed3eb36a107d55613a24186398d037825976eed44707cd"} Aug 01 00:28:53.836901 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.836882 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-69l6n" event={"ID":"cd7a3121-0e51-4f13-8d05-c0b10f115420","Type":"ContainerStarted","Data":"b63f6aa351c51f25b469c5da52e5245e9bfe9c27d42642b99ee494463a5af572"} Aug 01 00:28:53.838348 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.838328 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8w9d6" event={"ID":"fee8d179-f489-4751-895d-b5294aa866fb","Type":"ContainerStarted","Data":"1208228b1ac6f815238a5fa399d70b1d7aa6472c8ea767d19b3b34c6359d0f70"} Aug 01 00:28:53.848347 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.848309 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sbb29" podStartSLOduration=1.660966727 podStartE2EDuration="20.848297627s" totalImagesPullingTime="19.1873309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:28:53.847575024 +0000 UTC m=+22.636296870" watchObservedRunningTime="2026-08-01 00:28:53.848297627 +0000 UTC m=+22.637019452" Aug 01 00:28:53.875257 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.875207 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-195.ec2.internal" podStartSLOduration=19.875188022 podStartE2EDuration="19.875188022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:28:53.862001566 +0000 UTC m=+22.650723391" watchObservedRunningTime="2026-08-01 00:28:53.875188022 +0000 UTC m=+22.663909844" Aug 01 00:28:53.875712 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.875675 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8w9d6" podStartSLOduration=2.591712556 podStartE2EDuration="21.875664146s" totalImagesPullingTime="19.28395159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:28:53.875136657 +0000 UTC m=+22.663858484" watchObservedRunningTime="2026-08-01 00:28:53.875664146 +0000 UTC m=+22.664385996" Aug 01 00:28:53.888513 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.888482 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5b74f" podStartSLOduration=2.612843041 podStartE2EDuration="21.888469331s" totalImagesPullingTime="19.27562629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:28:53.888271324 +0000 UTC m=+22.676993147" watchObservedRunningTime="2026-08-01 00:28:53.888469331 +0000 UTC m=+22.677191155" Aug 01 00:28:53.931017 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:53.930975 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-69l6n" podStartSLOduration=2.671676446 podStartE2EDuration="21.930959315s" totalImagesPullingTime="19.259282869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:28:53.910220255 +0000 UTC m=+22.698942087" watchObservedRunningTime="2026-08-01 00:28:53.930959315 +0000 UTC m=+22.719681179" Aug 01 00:28:54.721956 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:54.721928 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:54.722144 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:54.722024 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:54.841819 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:54.841784 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gx8gw" event={"ID":"b09eb022-6d18-49d8-a5ba-51765459875c","Type":"ContainerStarted","Data":"d8ac7a0141ba2d8a4f60cac69d0d644e4ba623656662b1685b22df93d9db2307"} Aug 01 00:28:54.843209 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:54.843183 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jl7q2" event={"ID":"d8fb48d9-8b28-4dda-81d3-1401d003e825","Type":"ContainerStarted","Data":"4027c5f2cabdd29786a8aaa3c5f2aad83d6b33b6804799384b3882bc44a8a130"} Aug 01 00:28:54.845770 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:54.845755 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:28:54.846128 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:54.846102 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" event={"ID":"e7e36f0b-4b54-4945-be93-63ac46767270","Type":"ContainerStarted","Data":"c222b721cde6aae4163ffc4277563fa40a6a1deb64a6deed383bf158539a3feb"} Aug 01 00:28:54.846210 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:54.846177 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" event={"ID":"e7e36f0b-4b54-4945-be93-63ac46767270","Type":"ContainerStarted","Data":"d9a3b0c0eb9fed9d18a0bd021a8f0b4a92746f41aea1d4ed50f70d854cfb8c29"} Aug 01 00:28:54.846247 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:54.846226 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" event={"ID":"e7e36f0b-4b54-4945-be93-63ac46767270","Type":"ContainerStarted","Data":"aaec7cc82be69913c321af29d63d4014e833723e2d21bcc784990cfcc5720c69"} Aug 01 00:28:54.858240 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:54.858206 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gx8gw" podStartSLOduration=1.8450707899999998 podStartE2EDuration="21.858194899s" totalImagesPullingTime="20.013124109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:28:54.857648756 +0000 UTC m=+23.646370582" watchObservedRunningTime="2026-08-01 00:28:54.858194899 +0000 UTC m=+23.646916723" Aug 01 00:28:54.870554 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:54.870523 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jl7q2" podStartSLOduration=3.595832233 podStartE2EDuration="22.870511713s" totalImagesPullingTime="19.27467948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:28:54.870284533 +0000 UTC m=+23.659006360" watchObservedRunningTime="2026-08-01 00:28:54.870511713 +0000 UTC m=+23.659233578" Aug 01 00:28:55.003292 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:55.003267 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 01 00:28:55.691861 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:55.691762 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-01T00:28:55.003287525Z","UUID":"85bc59c8-1848-4418-99eb-42d8981a0782","Handler":null,"Name":"","Endpoint":""} Aug 01 00:28:55.693757 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:55.693736 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 01 00:28:55.693890 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:55.693777 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 01 00:28:55.722174 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:55.722149 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:55.722344 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:55.722178 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:55.722344 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:55.722268 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:55.722460 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:55.722396 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:55.851193 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:55.851156 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" event={"ID":"efcb618a-0402-4b4b-95cf-44c48b62abc2","Type":"ContainerStarted","Data":"b9221259d0edd4b69b0de35483684d22789c5d7186c69c9ebcc16ffc2658e233"} Aug 01 00:28:56.722109 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:56.721944 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:56.722279 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:56.722179 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:56.855401 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:56.855367 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" event={"ID":"efcb618a-0402-4b4b-95cf-44c48b62abc2","Type":"ContainerStarted","Data":"51b6a10833e5a2d83b304b49a95ff1eb69907de377f899f4c2179847798ca947"} Aug 01 00:28:56.858105 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:56.858087 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:28:56.858457 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:56.858438 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" event={"ID":"e7e36f0b-4b54-4945-be93-63ac46767270","Type":"ContainerStarted","Data":"7773ea5e07853fefb1751e69412278ced98c8d2fc874d8714c417454eeefd0db"} Aug 01 00:28:56.871578 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:56.871539 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzcj7" podStartSLOduration=1.849672591 podStartE2EDuration="23.871525035s" totalImagesPullingTime="22.021852444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:28:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:28:56.870022136 +0000 UTC m=+25.658743960" watchObservedRunningTime="2026-08-01 00:28:56.871525035 +0000 UTC m=+25.660246861" Aug 01 00:28:57.465281 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:57.465249 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sbb29" Aug 01 00:28:57.465858 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:57.465840 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sbb29" Aug 01 00:28:57.722203 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:57.722135 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:57.722354 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:57.722251 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:57.722354 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:57.722287 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:57.722470 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:57.722388 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:57.861102 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:57.861029 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sbb29" Aug 01 00:28:57.861628 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:57.861418 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sbb29" Aug 01 00:28:58.722352 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:58.722316 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:28:58.722518 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:58.722450 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:28:59.722416 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:59.722255 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:28:59.722921 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:59.722255 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:28:59.722921 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:59.722501 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:28:59.722921 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:28:59.722638 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:28:59.869924 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:59.869904 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:28:59.870258 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:59.870235 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" event={"ID":"e7e36f0b-4b54-4945-be93-63ac46767270","Type":"ContainerStarted","Data":"a3d65d9e48f343b770fc04dffa700f5bf96e0324e7472284c2d3eeebdf0e768e"} Aug 01 00:28:59.870569 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:59.870549 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:28:59.870666 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:59.870644 2563 scope.go:117] "RemoveContainer" containerID="1f5d8924458db543e751c4837adb9b6acd4ba4e8a441294ab16129d865f3b5bd" Aug 01 00:28:59.872136 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:59.872114 2563 generic.go:356] "Generic (PLEG): container finished" podID="a4e452e0-a47b-4d14-8852-17f8567c98fc" containerID="f154c7d959dccf53a39320614a0fa37acb4be65067291ca3e9cde835d6c6f2f8" exitCode=0 Aug 01 00:28:59.872221 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:59.872199 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q2wgc" event={"ID":"a4e452e0-a47b-4d14-8852-17f8567c98fc","Type":"ContainerDied","Data":"f154c7d959dccf53a39320614a0fa37acb4be65067291ca3e9cde835d6c6f2f8"} Aug 01 00:28:59.885555 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:28:59.885540 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:29:00.722907 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:00.722880 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:29:00.723330 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:00.722998 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:29:00.879868 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:00.879844 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:29:00.880215 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:00.880189 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" event={"ID":"e7e36f0b-4b54-4945-be93-63ac46767270","Type":"ContainerStarted","Data":"b0156527648c169c5f3fb497046e0038d807fcdf570ed96cbea5a6944b39f87c"} Aug 01 00:29:00.880470 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:00.880448 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:29:00.880584 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:00.880525 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:29:00.894594 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:00.894573 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:29:00.909169 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:00.909133 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" podStartSLOduration=8.403162981 podStartE2EDuration="27.90912019s" totalImagesPullingTime="19.505957209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:00.907366861 +0000 UTC m=+29.696088686" watchObservedRunningTime="2026-08-01 00:29:00.90912019 +0000 UTC m=+29.697842015" Aug 01 00:29:01.401842 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:01.401762 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wkj56"] Aug 01 00:29:01.401992 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:01.401905 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:29:01.402073 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:01.401998 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:29:01.404746 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:01.404720 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z87nn"] Aug 01 00:29:01.404877 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:01.404849 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:29:01.404964 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:01.404944 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:29:01.405464 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:01.405428 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nlw9l"] Aug 01 00:29:01.405579 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:01.405525 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:29:01.405634 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:01.405614 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:29:01.884157 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:01.884125 2563 generic.go:356] "Generic (PLEG): container finished" podID="a4e452e0-a47b-4d14-8852-17f8567c98fc" containerID="6f70eb8174f5b6c114462644eafa9c39c2b5ee509c0fe292309d9d61ccbb7995" exitCode=0 Aug 01 00:29:01.884565 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:01.884207 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q2wgc" event={"ID":"a4e452e0-a47b-4d14-8852-17f8567c98fc","Type":"ContainerDied","Data":"6f70eb8174f5b6c114462644eafa9c39c2b5ee509c0fe292309d9d61ccbb7995"} Aug 01 00:29:02.722640 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:02.722573 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:29:02.722787 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:02.722675 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:29:03.722726 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:03.722562 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:29:03.723355 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:03.722634 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:29:03.723355 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:03.722812 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlw9l" podUID="df497cda-451a-4e65-b0a8-b18769d3c9d2" Aug 01 00:29:03.723355 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:03.722899 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkj56" podUID="8644d164-f68a-4c37-abd5-cc4f66cbe590" Aug 01 00:29:03.891905 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:03.891869 2563 generic.go:356] "Generic (PLEG): container finished" podID="a4e452e0-a47b-4d14-8852-17f8567c98fc" containerID="d58d9ed7d68e434cc4060e55c357b9a6ccaaf9451fe3ccaca9e9a7d647be3e3b" exitCode=0 Aug 01 00:29:03.892065 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:03.891913 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q2wgc" event={"ID":"a4e452e0-a47b-4d14-8852-17f8567c98fc","Type":"ContainerDied","Data":"d58d9ed7d68e434cc4060e55c357b9a6ccaaf9451fe3ccaca9e9a7d647be3e3b"} Aug 01 00:29:04.722036 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:04.722003 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:29:04.722213 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:04.722129 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z87nn" podUID="225ac806-8d39-42b4-91ff-951ed22068f7" Aug 01 00:29:05.388752 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.388720 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:29:05.389231 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.388787 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqz4v\" (UniqueName: \"kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v\") pod \"network-check-target-z87nn\" (UID: \"225ac806-8d39-42b4-91ff-951ed22068f7\") " pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:29:05.389231 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:05.388890 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:05.389231 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:05.388915 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:29:05.389231 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:05.388932 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:29:05.389231 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:05.388944 2563 projected.go:196] Error preparing data for projected volume kube-api-access-kqz4v for pod openshift-network-diagnostics/network-check-target-z87nn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:05.389231 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:05.388968 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs podName:8644d164-f68a-4c37-abd5-cc4f66cbe590 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:37.388946172 +0000 UTC m=+66.177667994 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs") pod "network-metrics-daemon-wkj56" (UID: "8644d164-f68a-4c37-abd5-cc4f66cbe590") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:05.389231 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:05.388996 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v podName:225ac806-8d39-42b4-91ff-951ed22068f7 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:37.388980371 +0000 UTC m=+66.177702178 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kqz4v" (UniqueName: "kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v") pod "network-check-target-z87nn" (UID: "225ac806-8d39-42b4-91ff-951ed22068f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:05.489373 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.489340 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:29:05.489517 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:05.489460 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:05.489517 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:05.489506 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret podName:df497cda-451a-4e65-b0a8-b18769d3c9d2 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:37.489493083 +0000 UTC m=+66.278214887 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret") pod "global-pull-secret-syncer-nlw9l" (UID: "df497cda-451a-4e65-b0a8-b18769d3c9d2") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:05.548524 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.548500 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-195.ec2.internal" event="NodeReady" Aug 01 00:29:05.548647 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.548618 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 01 00:29:05.580552 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.580520 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fcd8946f6-l8kx8"] Aug 01 00:29:05.582407 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.582382 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.585218 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.585044 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 01 00:29:05.585218 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.585098 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4jrsr\"" Aug 01 00:29:05.585218 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.585044 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 01 00:29:05.585218 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.585168 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 01 00:29:05.590644 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.590569 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 01 00:29:05.596083 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.595959 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fcd8946f6-l8kx8"] Aug 01 00:29:05.600031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.599637 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5kstw"] Aug 01 00:29:05.602571 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.602549 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d7hj6"] Aug 01 00:29:05.602692 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.602673 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5kstw" Aug 01 00:29:05.604466 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.604447 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:05.604973 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.604954 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 01 00:29:05.605129 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.605109 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 01 00:29:05.605275 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.605258 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-stmw4\"" Aug 01 00:29:05.605398 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.605193 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 01 00:29:05.606559 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.606542 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 01 00:29:05.606679 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.606542 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fl89b\"" Aug 01 00:29:05.606795 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.606578 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 01 00:29:05.609223 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.609178 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5kstw"] Aug 01 00:29:05.612399 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.612378 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7hj6"] Aug 01 00:29:05.690497 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690422 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.690497 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690454 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5dc1c37-a456-45c8-a01c-c30082421e3b-installation-pull-secrets\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.690497 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690486 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a6ef2cf-9383-4287-8b3d-5fc395998c22-cert\") pod \"ingress-canary-5kstw\" (UID: \"1a6ef2cf-9383-4287-8b3d-5fc395998c22\") " pod="openshift-ingress-canary/ingress-canary-5kstw" Aug 01 00:29:05.690734 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690541 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t89nf\" (UniqueName: \"kubernetes.io/projected/1a6ef2cf-9383-4287-8b3d-5fc395998c22-kube-api-access-t89nf\") pod \"ingress-canary-5kstw\" (UID: \"1a6ef2cf-9383-4287-8b3d-5fc395998c22\") " pod="openshift-ingress-canary/ingress-canary-5kstw" Aug 01 00:29:05.690734 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690573 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-bound-sa-token\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.690734 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690594 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:05.690734 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690617 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz5bx\" (UniqueName: \"kubernetes.io/projected/e5b71018-3dd9-43cc-88e1-220d978adcf6-kube-api-access-tz5bx\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:05.690734 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690664 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5dc1c37-a456-45c8-a01c-c30082421e3b-image-registry-private-configuration\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.690734 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690696 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5dc1c37-a456-45c8-a01c-c30082421e3b-trusted-ca\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.690734 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690727 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b71018-3dd9-43cc-88e1-220d978adcf6-tmp-dir\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:05.691195 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690814 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-certificates\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.691195 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690864 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5dc1c37-a456-45c8-a01c-c30082421e3b-ca-trust-extracted\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.691195 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690920 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c5ns\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-kube-api-access-4c5ns\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.691195 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.690947 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b71018-3dd9-43cc-88e1-220d978adcf6-config-volume\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:05.723028 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.722995 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:29:05.723245 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.723228 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:29:05.725906 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.725883 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 01 00:29:05.726014 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.725886 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c2x5h\"" Aug 01 00:29:05.726087 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.725887 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 01 00:29:05.731361 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.731340 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:05.791283 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791250 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5dc1c37-a456-45c8-a01c-c30082421e3b-image-registry-private-configuration\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.791283 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791282 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5dc1c37-a456-45c8-a01c-c30082421e3b-trusted-ca\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.791477 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b71018-3dd9-43cc-88e1-220d978adcf6-tmp-dir\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:05.791477 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791342 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-certificates\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.791477 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791369 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5dc1c37-a456-45c8-a01c-c30082421e3b-ca-trust-extracted\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.791477 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791430 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c5ns\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-kube-api-access-4c5ns\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.791477 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791469 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b71018-3dd9-43cc-88e1-220d978adcf6-config-volume\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:05.791704 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791535 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.791704 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791561 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5dc1c37-a456-45c8-a01c-c30082421e3b-installation-pull-secrets\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.791704 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791596 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a6ef2cf-9383-4287-8b3d-5fc395998c22-cert\") pod \"ingress-canary-5kstw\" (UID: \"1a6ef2cf-9383-4287-8b3d-5fc395998c22\") " pod="openshift-ingress-canary/ingress-canary-5kstw" Aug 01 00:29:05.791704 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791619 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t89nf\" (UniqueName: \"kubernetes.io/projected/1a6ef2cf-9383-4287-8b3d-5fc395998c22-kube-api-access-t89nf\") pod \"ingress-canary-5kstw\" (UID: \"1a6ef2cf-9383-4287-8b3d-5fc395998c22\") " pod="openshift-ingress-canary/ingress-canary-5kstw" Aug 01 00:29:05.791704 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791645 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-bound-sa-token\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.791704 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791666 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:05.791704 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791689 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz5bx\" (UniqueName: \"kubernetes.io/projected/e5b71018-3dd9-43cc-88e1-220d978adcf6-kube-api-access-tz5bx\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:05.792047 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.791728 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b71018-3dd9-43cc-88e1-220d978adcf6-tmp-dir\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:05.792047 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:05.791819 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:29:05.792047 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:05.791928 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls podName:e5b71018-3dd9-43cc-88e1-220d978adcf6 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:06.291909986 +0000 UTC m=+35.080631794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls") pod "dns-default-d7hj6" (UID: "e5b71018-3dd9-43cc-88e1-220d978adcf6") : secret "dns-default-metrics-tls" not found Aug 01 00:29:05.792047 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:05.791954 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:29:05.792047 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:05.791972 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcd8946f6-l8kx8: secret "image-registry-tls" not found Aug 01 00:29:05.792047 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:05.792042 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls podName:e5dc1c37-a456-45c8-a01c-c30082421e3b nodeName:}" failed. No retries permitted until 2026-08-01 00:29:06.292026005 +0000 UTC m=+35.080747850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls") pod "image-registry-6fcd8946f6-l8kx8" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b") : secret "image-registry-tls" not found Aug 01 00:29:05.792356 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.792282 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-certificates\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.792356 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.792338 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5dc1c37-a456-45c8-a01c-c30082421e3b-trusted-ca\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.792473 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.792449 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5dc1c37-a456-45c8-a01c-c30082421e3b-ca-trust-extracted\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.792595 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.792509 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b71018-3dd9-43cc-88e1-220d978adcf6-config-volume\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:05.795623 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.795604 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5dc1c37-a456-45c8-a01c-c30082421e3b-image-registry-private-configuration\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.795728 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.795636 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a6ef2cf-9383-4287-8b3d-5fc395998c22-cert\") pod \"ingress-canary-5kstw\" (UID: \"1a6ef2cf-9383-4287-8b3d-5fc395998c22\") " pod="openshift-ingress-canary/ingress-canary-5kstw" Aug 01 00:29:05.795728 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.795677 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5dc1c37-a456-45c8-a01c-c30082421e3b-installation-pull-secrets\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.801286 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.801266 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz5bx\" (UniqueName: \"kubernetes.io/projected/e5b71018-3dd9-43cc-88e1-220d978adcf6-kube-api-access-tz5bx\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:05.802415 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.802395 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t89nf\" (UniqueName: \"kubernetes.io/projected/1a6ef2cf-9383-4287-8b3d-5fc395998c22-kube-api-access-t89nf\") pod \"ingress-canary-5kstw\" (UID: \"1a6ef2cf-9383-4287-8b3d-5fc395998c22\") " pod="openshift-ingress-canary/ingress-canary-5kstw" Aug 01 00:29:05.802686 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.802665 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-bound-sa-token\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.802799 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.802784 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c5ns\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-kube-api-access-4c5ns\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:05.815526 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.815432 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:05.914381 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:05.914356 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5kstw" Aug 01 00:29:06.062537 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:06.062360 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5kstw"] Aug 01 00:29:06.066085 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:29:06.066046 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6ef2cf_9383_4287_8b3d_5fc395998c22.slice/crio-422631d508769b16cb47d3105be2329ba5bc9d3ca317d3f527c2bae3e129f058 WatchSource:0}: Error finding container 422631d508769b16cb47d3105be2329ba5bc9d3ca317d3f527c2bae3e129f058: Status 404 returned error can't find the container with id 422631d508769b16cb47d3105be2329ba5bc9d3ca317d3f527c2bae3e129f058 Aug 01 00:29:06.295737 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:06.295629 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:06.295737 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:06.295686 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:06.295954 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:06.295785 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:29:06.295954 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:06.295803 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcd8946f6-l8kx8: secret "image-registry-tls" not found Aug 01 00:29:06.295954 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:06.295862 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls podName:e5dc1c37-a456-45c8-a01c-c30082421e3b nodeName:}" failed. No retries permitted until 2026-08-01 00:29:07.295842195 +0000 UTC m=+36.084564004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls") pod "image-registry-6fcd8946f6-l8kx8" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b") : secret "image-registry-tls" not found Aug 01 00:29:06.295954 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:06.295869 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:29:06.295954 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:06.295930 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls podName:e5b71018-3dd9-43cc-88e1-220d978adcf6 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:07.295909988 +0000 UTC m=+36.084631795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls") pod "dns-default-d7hj6" (UID: "e5b71018-3dd9-43cc-88e1-220d978adcf6") : secret "dns-default-metrics-tls" not found Aug 01 00:29:06.722442 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:06.722287 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:29:06.725278 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:06.725041 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 01 00:29:06.725278 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:06.725074 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bnzmh\"" Aug 01 00:29:06.725278 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:06.725191 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 01 00:29:06.899761 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:06.899733 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5kstw" event={"ID":"1a6ef2cf-9383-4287-8b3d-5fc395998c22","Type":"ContainerStarted","Data":"422631d508769b16cb47d3105be2329ba5bc9d3ca317d3f527c2bae3e129f058"} Aug 01 00:29:07.302336 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:07.302299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:07.302522 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:07.302357 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:07.302522 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:07.302489 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:29:07.302522 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:07.302518 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcd8946f6-l8kx8: secret "image-registry-tls" not found Aug 01 00:29:07.302681 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:07.302502 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:29:07.302681 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:07.302581 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls podName:e5dc1c37-a456-45c8-a01c-c30082421e3b nodeName:}" failed. No retries permitted until 2026-08-01 00:29:09.302562337 +0000 UTC m=+38.091284152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls") pod "image-registry-6fcd8946f6-l8kx8" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b") : secret "image-registry-tls" not found Aug 01 00:29:07.302681 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:07.302656 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls podName:e5b71018-3dd9-43cc-88e1-220d978adcf6 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:09.302637327 +0000 UTC m=+38.091359133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls") pod "dns-default-d7hj6" (UID: "e5b71018-3dd9-43cc-88e1-220d978adcf6") : secret "dns-default-metrics-tls" not found Aug 01 00:29:09.318686 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:09.318646 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:09.319153 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:09.318706 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:09.319153 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:09.318847 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:29:09.319153 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:09.318874 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcd8946f6-l8kx8: secret "image-registry-tls" not found Aug 01 00:29:09.319153 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:09.318931 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls podName:e5dc1c37-a456-45c8-a01c-c30082421e3b nodeName:}" failed. No retries permitted until 2026-08-01 00:29:13.318915062 +0000 UTC m=+42.107636866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls") pod "image-registry-6fcd8946f6-l8kx8" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b") : secret "image-registry-tls" not found Aug 01 00:29:09.319153 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:09.318850 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:29:09.319153 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:09.318977 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls podName:e5b71018-3dd9-43cc-88e1-220d978adcf6 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:13.318968716 +0000 UTC m=+42.107690523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls") pod "dns-default-d7hj6" (UID: "e5b71018-3dd9-43cc-88e1-220d978adcf6") : secret "dns-default-metrics-tls" not found Aug 01 00:29:10.307134 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:10.307102 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:10.357642 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:10.357614 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:10.911630 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:10.911601 2563 generic.go:356] "Generic (PLEG): container finished" podID="a4e452e0-a47b-4d14-8852-17f8567c98fc" containerID="e03dc0d8f49899cb25f25c9a2d44f2610c474dbd10cb68d31ff8a04255c54911" exitCode=0 Aug 01 00:29:10.911776 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:10.911687 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q2wgc" event={"ID":"a4e452e0-a47b-4d14-8852-17f8567c98fc","Type":"ContainerDied","Data":"e03dc0d8f49899cb25f25c9a2d44f2610c474dbd10cb68d31ff8a04255c54911"} Aug 01 00:29:10.913150 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:10.913129 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5kstw" event={"ID":"1a6ef2cf-9383-4287-8b3d-5fc395998c22","Type":"ContainerStarted","Data":"21433c70bb1aa5a6632be1218f1a7a192edd36c5c184e99d260d4b49a60b664c"} Aug 01 00:29:10.944604 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:10.944531 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5kstw" podStartSLOduration=2.153084152 podStartE2EDuration="5.944513908s" totalImagesPullingTime="3.791429756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:10.944462955 +0000 UTC m=+39.733184780" watchObservedRunningTime="2026-08-01 00:29:10.944513908 +0000 UTC m=+39.733235733" Aug 01 00:29:11.529622 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:11.529445 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:11.629791 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:11.629751 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:11.704356 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:11.704315 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:11.807894 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:11.807861 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:11.906183 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:11.906142 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:11.918388 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:11.918358 2563 generic.go:356] "Generic (PLEG): container finished" podID="a4e452e0-a47b-4d14-8852-17f8567c98fc" containerID="48c3bfaeabeb14203ebc9054b4288c9fd21c659223473cca24888f80606d2428" exitCode=0 Aug 01 00:29:11.918518 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:11.918448 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q2wgc" event={"ID":"a4e452e0-a47b-4d14-8852-17f8567c98fc","Type":"ContainerDied","Data":"48c3bfaeabeb14203ebc9054b4288c9fd21c659223473cca24888f80606d2428"} Aug 01 00:29:12.007232 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:12.007208 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:12.485706 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:12.485603 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:12.544888 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:12.544855 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:12.889827 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:12.889787 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:12.923431 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:12.923395 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q2wgc" event={"ID":"a4e452e0-a47b-4d14-8852-17f8567c98fc","Type":"ContainerStarted","Data":"c70ceff72bc52ab454aeb15b3a75d73177dec02faec0ef4e5a91a018aeac5413"} Aug 01 00:29:12.947781 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:12.947739 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q2wgc" podStartSLOduration=7.751410877 podStartE2EDuration="40.947724162s" totalImagesPullingTime="33.196313285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:28:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:12.946601133 +0000 UTC m=+41.735322989" watchObservedRunningTime="2026-08-01 00:29:12.947724162 +0000 UTC m=+41.736445983" Aug 01 00:29:12.950744 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:12.950714 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:13.350386 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:13.350355 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:13.350539 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:13.350395 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:13.350539 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:13.350512 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:29:13.350539 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:13.350529 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcd8946f6-l8kx8: secret "image-registry-tls" not found Aug 01 00:29:13.350637 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:13.350577 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls podName:e5dc1c37-a456-45c8-a01c-c30082421e3b nodeName:}" failed. No retries permitted until 2026-08-01 00:29:21.350563391 +0000 UTC m=+50.139285197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls") pod "image-registry-6fcd8946f6-l8kx8" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b") : secret "image-registry-tls" not found Aug 01 00:29:13.350637 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:13.350517 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:29:13.350637 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:13.350629 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls podName:e5b71018-3dd9-43cc-88e1-220d978adcf6 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:21.350619655 +0000 UTC m=+50.139341458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls") pod "dns-default-d7hj6" (UID: "e5b71018-3dd9-43cc-88e1-220d978adcf6") : secret "dns-default-metrics-tls" not found Aug 01 00:29:13.682418 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:13.682332 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:13.729853 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:13.729821 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:14.084663 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:14.084630 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:14.146283 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:14.146248 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:14.245306 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:14.245270 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:14.877913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:14.877873 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:14.983399 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:14.983366 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:15.787510 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:15.787478 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:15.888436 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:15.888403 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:16.209221 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:16.209178 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:16.480232 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:16.480124 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:16.580769 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:16.580739 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:19.171791 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:19.171757 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:21.408368 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:21.408332 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:21.408760 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:21.408387 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:21.408760 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:21.408515 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:29:21.408760 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:21.408542 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcd8946f6-l8kx8: secret "image-registry-tls" not found Aug 01 00:29:21.408760 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:21.408552 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:29:21.408760 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:21.408612 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls podName:e5dc1c37-a456-45c8-a01c-c30082421e3b nodeName:}" failed. No retries permitted until 2026-08-01 00:29:37.40859155 +0000 UTC m=+66.197313360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls") pod "image-registry-6fcd8946f6-l8kx8" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b") : secret "image-registry-tls" not found Aug 01 00:29:21.408760 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:21.408632 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls podName:e5b71018-3dd9-43cc-88e1-220d978adcf6 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:37.408622711 +0000 UTC m=+66.197344557 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls") pod "dns-default-d7hj6" (UID: "e5b71018-3dd9-43cc-88e1-220d978adcf6") : secret "dns-default-metrics-tls" not found Aug 01 00:29:26.760339 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:26.760307 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:31.140896 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:31.140861 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:31.242535 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:31.242502 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:32.896865 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:32.896835 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b54j8" Aug 01 00:29:33.048599 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:33.048555 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:33.148074 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:33.147971 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:33.201839 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:33.201803 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:37.424560 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.424520 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:29:37.424956 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.424568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqz4v\" (UniqueName: \"kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v\") pod \"network-check-target-z87nn\" (UID: \"225ac806-8d39-42b4-91ff-951ed22068f7\") " pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:29:37.424956 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.424607 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:29:37.424956 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.424624 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:29:37.424956 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:37.424676 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:29:37.424956 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:37.424701 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:29:37.424956 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:37.424711 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcd8946f6-l8kx8: secret "image-registry-tls" not found Aug 01 00:29:37.424956 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:37.424754 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls podName:e5b71018-3dd9-43cc-88e1-220d978adcf6 nodeName:}" failed. No retries permitted until 2026-08-01 00:30:09.424738677 +0000 UTC m=+98.213460482 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls") pod "dns-default-d7hj6" (UID: "e5b71018-3dd9-43cc-88e1-220d978adcf6") : secret "dns-default-metrics-tls" not found Aug 01 00:29:37.424956 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:37.424767 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls podName:e5dc1c37-a456-45c8-a01c-c30082421e3b nodeName:}" failed. No retries permitted until 2026-08-01 00:30:09.424761427 +0000 UTC m=+98.213483230 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls") pod "image-registry-6fcd8946f6-l8kx8" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b") : secret "image-registry-tls" not found Aug 01 00:29:37.431336 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.431310 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 01 00:29:37.431336 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.431326 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 01 00:29:37.435718 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:37.435701 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 01 00:29:37.435770 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:37.435763 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs podName:8644d164-f68a-4c37-abd5-cc4f66cbe590 nodeName:}" failed. No retries permitted until 2026-08-01 00:30:41.435744609 +0000 UTC m=+130.224466413 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs") pod "network-metrics-daemon-wkj56" (UID: "8644d164-f68a-4c37-abd5-cc4f66cbe590") : secret "metrics-daemon-secret" not found Aug 01 00:29:37.437550 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.437534 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 01 00:29:37.449591 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.449563 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqz4v\" (UniqueName: \"kubernetes.io/projected/225ac806-8d39-42b4-91ff-951ed22068f7-kube-api-access-kqz4v\") pod \"network-check-target-z87nn\" (UID: \"225ac806-8d39-42b4-91ff-951ed22068f7\") " pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:29:37.525881 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.525829 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:29:37.528607 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.528591 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 01 00:29:37.538662 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.538639 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df497cda-451a-4e65-b0a8-b18769d3c9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nlw9l\" (UID: \"df497cda-451a-4e65-b0a8-b18769d3c9d2\") " pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:29:37.636449 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.636415 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bnzmh\"" Aug 01 00:29:37.644428 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.644407 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:29:37.759558 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.759530 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z87nn"] Aug 01 00:29:37.762440 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:29:37.762417 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225ac806_8d39_42b4_91ff_951ed22068f7.slice/crio-818f787b68808b35928c29fa4820f5c936f90fdc8b626f7e19384d24c91681b4 WatchSource:0}: Error finding container 818f787b68808b35928c29fa4820f5c936f90fdc8b626f7e19384d24c91681b4: Status 404 returned error can't find the container with id 818f787b68808b35928c29fa4820f5c936f90fdc8b626f7e19384d24c91681b4 Aug 01 00:29:37.833858 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.833825 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlw9l" Aug 01 00:29:37.948129 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.948041 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nlw9l"] Aug 01 00:29:37.951312 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:29:37.951284 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf497cda_451a_4e65_b0a8_b18769d3c9d2.slice/crio-b5fd8f400711319a1450950ec1d3d61eee69f86e2f03104adf73f9b8981d4f81 WatchSource:0}: Error finding container b5fd8f400711319a1450950ec1d3d61eee69f86e2f03104adf73f9b8981d4f81: Status 404 returned error can't find the container with id b5fd8f400711319a1450950ec1d3d61eee69f86e2f03104adf73f9b8981d4f81 Aug 01 00:29:37.980743 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.980707 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nlw9l" event={"ID":"df497cda-451a-4e65-b0a8-b18769d3c9d2","Type":"ContainerStarted","Data":"b5fd8f400711319a1450950ec1d3d61eee69f86e2f03104adf73f9b8981d4f81"} Aug 01 00:29:37.981731 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:37.981703 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z87nn" event={"ID":"225ac806-8d39-42b4-91ff-951ed22068f7","Type":"ContainerStarted","Data":"818f787b68808b35928c29fa4820f5c936f90fdc8b626f7e19384d24c91681b4"} Aug 01 00:29:38.397881 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.397842 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh"] Aug 01 00:29:38.411004 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.410977 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh"] Aug 01 00:29:38.411191 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.411112 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:38.413691 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.413666 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 01 00:29:38.415043 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.415020 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 01 00:29:38.415179 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.415150 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hmvq9\"" Aug 01 00:29:38.415887 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.415863 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Aug 01 00:29:38.416142 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.416125 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Aug 01 00:29:38.432736 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.432706 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:38.433128 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.432756 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:38.433128 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.432829 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msvcf\" (UniqueName: \"kubernetes.io/projected/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-kube-api-access-msvcf\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:38.500292 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.500258 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh"] Aug 01 00:29:38.522891 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.522857 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k"] Aug 01 00:29:38.523068 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.522975 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:29:38.525253 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.525229 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:29:38.525253 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.525246 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-d4dlk\"" Aug 01 00:29:38.525253 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.525235 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Aug 01 00:29:38.525479 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.525318 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Aug 01 00:29:38.533473 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.533449 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:38.533604 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.533494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msvcf\" (UniqueName: \"kubernetes.io/projected/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-kube-api-access-msvcf\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:38.533604 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.533534 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-t2tnh\" (UID: \"209f7a64-59e3-4d72-9b5a-22f72f01e2dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:29:38.533604 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.533559 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fdl8\" (UniqueName: \"kubernetes.io/projected/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-kube-api-access-8fdl8\") pod \"cluster-samples-operator-559cbf5745-t2tnh\" (UID: \"209f7a64-59e3-4d72-9b5a-22f72f01e2dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:29:38.533763 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.533737 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:38.533846 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:38.533833 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 01 00:29:38.533897 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:38.533889 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls podName:802cbf7b-10e0-4ff6-8443-c0f7b60a9727 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:39.033874832 +0000 UTC m=+67.822596640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-znmsh" (UID: "802cbf7b-10e0-4ff6-8443-c0f7b60a9727") : secret "cluster-monitoring-operator-tls" not found Aug 01 00:29:38.534114 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.534097 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:38.545035 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.545005 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh"] Aug 01 00:29:38.545149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.545048 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k"] Aug 01 00:29:38.545149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.545123 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" Aug 01 00:29:38.548027 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.548007 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 01 00:29:38.548027 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.548025 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-j2fd4\"" Aug 01 00:29:38.548208 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.548042 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 01 00:29:38.548358 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.548340 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 01 00:29:38.548426 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.548350 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:29:38.554896 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.554875 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msvcf\" (UniqueName: \"kubernetes.io/projected/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-kube-api-access-msvcf\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:38.597167 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.597131 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-nlksz"] Aug 01 00:29:38.611249 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.611219 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-nlksz"] Aug 01 00:29:38.611415 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.611318 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.613959 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.613932 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Aug 01 00:29:38.614139 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.613935 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 01 00:29:38.614139 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.613976 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 01 00:29:38.614139 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.613997 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-977kn\"" Aug 01 00:29:38.614315 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.614231 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Aug 01 00:29:38.645842 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.645801 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb4vn\" (UniqueName: \"kubernetes.io/projected/bcacc184-28ea-4d69-a87c-b7c08c2add99-kube-api-access-nb4vn\") pod \"service-ca-operator-784f97bcf6-6fc4k\" (UID: \"bcacc184-28ea-4d69-a87c-b7c08c2add99\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" Aug 01 00:29:38.646041 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.645861 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a878cee9-835b-4d43-9d83-b5469f709610-tmp\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.646041 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.645898 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-t2tnh\" (UID: \"209f7a64-59e3-4d72-9b5a-22f72f01e2dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:29:38.646041 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.645925 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bcacc184-28ea-4d69-a87c-b7c08c2add99-serving-cert\") pod \"service-ca-operator-784f97bcf6-6fc4k\" (UID: \"bcacc184-28ea-4d69-a87c-b7c08c2add99\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" Aug 01 00:29:38.646041 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.645971 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a878cee9-835b-4d43-9d83-b5469f709610-snapshots\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.646041 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:38.646031 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 01 00:29:38.646302 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.646039 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a878cee9-835b-4d43-9d83-b5469f709610-trusted-ca-bundle\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.646302 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:38.646110 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls podName:209f7a64-59e3-4d72-9b5a-22f72f01e2dc nodeName:}" failed. No retries permitted until 2026-08-01 00:29:39.146089984 +0000 UTC m=+67.934811803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-t2tnh" (UID: "209f7a64-59e3-4d72-9b5a-22f72f01e2dc") : secret "samples-operator-tls" not found Aug 01 00:29:38.646302 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.646126 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a878cee9-835b-4d43-9d83-b5469f709610-serving-cert\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.646302 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.646171 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fdl8\" (UniqueName: \"kubernetes.io/projected/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-kube-api-access-8fdl8\") pod \"cluster-samples-operator-559cbf5745-t2tnh\" (UID: \"209f7a64-59e3-4d72-9b5a-22f72f01e2dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:29:38.646302 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.646198 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9666l\" (UniqueName: \"kubernetes.io/projected/a878cee9-835b-4d43-9d83-b5469f709610-kube-api-access-9666l\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.646302 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.646247 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bcacc184-28ea-4d69-a87c-b7c08c2add99-config\") pod \"service-ca-operator-784f97bcf6-6fc4k\" (UID: \"bcacc184-28ea-4d69-a87c-b7c08c2add99\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" Aug 01 00:29:38.646302 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.646270 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a878cee9-835b-4d43-9d83-b5469f709610-service-ca-bundle\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.653534 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.653470 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Aug 01 00:29:38.655525 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.655503 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fdl8\" (UniqueName: \"kubernetes.io/projected/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-kube-api-access-8fdl8\") pod \"cluster-samples-operator-559cbf5745-t2tnh\" (UID: \"209f7a64-59e3-4d72-9b5a-22f72f01e2dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:29:38.747450 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.747400 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a878cee9-835b-4d43-9d83-b5469f709610-serving-cert\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.747617 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.747468 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9666l\" (UniqueName: \"kubernetes.io/projected/a878cee9-835b-4d43-9d83-b5469f709610-kube-api-access-9666l\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.747617 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.747525 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bcacc184-28ea-4d69-a87c-b7c08c2add99-config\") pod \"service-ca-operator-784f97bcf6-6fc4k\" (UID: \"bcacc184-28ea-4d69-a87c-b7c08c2add99\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" Aug 01 00:29:38.747617 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.747542 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a878cee9-835b-4d43-9d83-b5469f709610-service-ca-bundle\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.747617 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.747570 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb4vn\" (UniqueName: \"kubernetes.io/projected/bcacc184-28ea-4d69-a87c-b7c08c2add99-kube-api-access-nb4vn\") pod \"service-ca-operator-784f97bcf6-6fc4k\" (UID: \"bcacc184-28ea-4d69-a87c-b7c08c2add99\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" Aug 01 00:29:38.747617 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.747598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a878cee9-835b-4d43-9d83-b5469f709610-tmp\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.747888 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.747624 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bcacc184-28ea-4d69-a87c-b7c08c2add99-serving-cert\") pod \"service-ca-operator-784f97bcf6-6fc4k\" (UID: \"bcacc184-28ea-4d69-a87c-b7c08c2add99\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" Aug 01 00:29:38.747888 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.747652 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a878cee9-835b-4d43-9d83-b5469f709610-snapshots\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.747888 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.747679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a878cee9-835b-4d43-9d83-b5469f709610-trusted-ca-bundle\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.748537 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.748510 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a878cee9-835b-4d43-9d83-b5469f709610-service-ca-bundle\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.748661 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.748623 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a878cee9-835b-4d43-9d83-b5469f709610-trusted-ca-bundle\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.748661 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.748632 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a878cee9-835b-4d43-9d83-b5469f709610-snapshots\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.748907 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.748884 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a878cee9-835b-4d43-9d83-b5469f709610-tmp\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.749033 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.749011 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bcacc184-28ea-4d69-a87c-b7c08c2add99-config\") pod \"service-ca-operator-784f97bcf6-6fc4k\" (UID: \"bcacc184-28ea-4d69-a87c-b7c08c2add99\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" Aug 01 00:29:38.750308 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.750283 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bcacc184-28ea-4d69-a87c-b7c08c2add99-serving-cert\") pod \"service-ca-operator-784f97bcf6-6fc4k\" (UID: \"bcacc184-28ea-4d69-a87c-b7c08c2add99\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" Aug 01 00:29:38.750421 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.750398 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a878cee9-835b-4d43-9d83-b5469f709610-serving-cert\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.757618 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.757467 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb4vn\" (UniqueName: \"kubernetes.io/projected/bcacc184-28ea-4d69-a87c-b7c08c2add99-kube-api-access-nb4vn\") pod \"service-ca-operator-784f97bcf6-6fc4k\" (UID: \"bcacc184-28ea-4d69-a87c-b7c08c2add99\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" Aug 01 00:29:38.758288 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.758244 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9666l\" (UniqueName: \"kubernetes.io/projected/a878cee9-835b-4d43-9d83-b5469f709610-kube-api-access-9666l\") pod \"insights-operator-77696685b-nlksz\" (UID: \"a878cee9-835b-4d43-9d83-b5469f709610\") " pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:38.853882 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.853836 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" Aug 01 00:29:38.924276 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:38.923997 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-nlksz" Aug 01 00:29:39.046014 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:39.045982 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k"] Aug 01 00:29:39.049567 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:29:39.049536 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcacc184_28ea_4d69_a87c_b7c08c2add99.slice/crio-c67dc3f262f6ec0d87b69c4d236cd7d8241fd259596531b4b9a2b3e1c8c38d4a WatchSource:0}: Error finding container c67dc3f262f6ec0d87b69c4d236cd7d8241fd259596531b4b9a2b3e1c8c38d4a: Status 404 returned error can't find the container with id c67dc3f262f6ec0d87b69c4d236cd7d8241fd259596531b4b9a2b3e1c8c38d4a Aug 01 00:29:39.051307 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:39.051200 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:39.051449 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:39.051429 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 01 00:29:39.051568 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:39.051551 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls podName:802cbf7b-10e0-4ff6-8443-c0f7b60a9727 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:40.051513809 +0000 UTC m=+68.840235620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-znmsh" (UID: "802cbf7b-10e0-4ff6-8443-c0f7b60a9727") : secret "cluster-monitoring-operator-tls" not found Aug 01 00:29:39.100750 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:39.100676 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-nlksz"] Aug 01 00:29:39.103892 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:29:39.103865 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda878cee9_835b_4d43_9d83_b5469f709610.slice/crio-ad44c6b0feaee6fdb313728ecb785e5fc7ebd1957d3361127bf56a34b9c9fd64 WatchSource:0}: Error finding container ad44c6b0feaee6fdb313728ecb785e5fc7ebd1957d3361127bf56a34b9c9fd64: Status 404 returned error can't find the container with id ad44c6b0feaee6fdb313728ecb785e5fc7ebd1957d3361127bf56a34b9c9fd64 Aug 01 00:29:39.153436 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:39.153396 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-t2tnh\" (UID: \"209f7a64-59e3-4d72-9b5a-22f72f01e2dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:29:39.153593 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:39.153562 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 01 00:29:39.153656 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:39.153628 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls podName:209f7a64-59e3-4d72-9b5a-22f72f01e2dc nodeName:}" failed. No retries permitted until 2026-08-01 00:29:40.15360847 +0000 UTC m=+68.942330282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-t2tnh" (UID: "209f7a64-59e3-4d72-9b5a-22f72f01e2dc") : secret "samples-operator-tls" not found Aug 01 00:29:39.996504 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:39.996353 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-nlksz" event={"ID":"a878cee9-835b-4d43-9d83-b5469f709610","Type":"ContainerStarted","Data":"ad44c6b0feaee6fdb313728ecb785e5fc7ebd1957d3361127bf56a34b9c9fd64"} Aug 01 00:29:39.998757 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:39.998681 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" event={"ID":"bcacc184-28ea-4d69-a87c-b7c08c2add99","Type":"ContainerStarted","Data":"c67dc3f262f6ec0d87b69c4d236cd7d8241fd259596531b4b9a2b3e1c8c38d4a"} Aug 01 00:29:40.060934 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:40.060895 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:40.061135 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:40.061109 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 01 00:29:40.061209 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:40.061181 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls podName:802cbf7b-10e0-4ff6-8443-c0f7b60a9727 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:42.061159689 +0000 UTC m=+70.849881506 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-znmsh" (UID: "802cbf7b-10e0-4ff6-8443-c0f7b60a9727") : secret "cluster-monitoring-operator-tls" not found Aug 01 00:29:40.162232 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:40.162117 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-t2tnh\" (UID: \"209f7a64-59e3-4d72-9b5a-22f72f01e2dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:29:40.162454 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:40.162237 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 01 00:29:40.162454 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:40.162325 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls podName:209f7a64-59e3-4d72-9b5a-22f72f01e2dc nodeName:}" failed. No retries permitted until 2026-08-01 00:29:42.162304269 +0000 UTC m=+70.951026087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-t2tnh" (UID: "209f7a64-59e3-4d72-9b5a-22f72f01e2dc") : secret "samples-operator-tls" not found Aug 01 00:29:42.077735 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:42.077694 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:42.078321 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:42.077866 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 01 00:29:42.078321 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:42.077951 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls podName:802cbf7b-10e0-4ff6-8443-c0f7b60a9727 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:46.077929123 +0000 UTC m=+74.866650930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-znmsh" (UID: "802cbf7b-10e0-4ff6-8443-c0f7b60a9727") : secret "cluster-monitoring-operator-tls" not found Aug 01 00:29:42.178603 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:42.178561 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-t2tnh\" (UID: \"209f7a64-59e3-4d72-9b5a-22f72f01e2dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:29:42.178775 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:42.178692 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 01 00:29:42.178775 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:42.178773 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls podName:209f7a64-59e3-4d72-9b5a-22f72f01e2dc nodeName:}" failed. No retries permitted until 2026-08-01 00:29:46.178751751 +0000 UTC m=+74.967473559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-t2tnh" (UID: "209f7a64-59e3-4d72-9b5a-22f72f01e2dc") : secret "samples-operator-tls" not found Aug 01 00:29:44.013302 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:44.013268 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" event={"ID":"bcacc184-28ea-4d69-a87c-b7c08c2add99","Type":"ContainerStarted","Data":"170e6104a95d8bfff9880e5fdcca87d3cab9b25da9cdb8f64985755eb1bf385c"} Aug 01 00:29:44.014942 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:44.014904 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nlw9l" event={"ID":"df497cda-451a-4e65-b0a8-b18769d3c9d2","Type":"ContainerStarted","Data":"b456fd7e34fbab5b5628343def05a58648b5113a00f31edf6cf97f0eeeab8a1d"} Aug 01 00:29:44.016352 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:44.016301 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z87nn" event={"ID":"225ac806-8d39-42b4-91ff-951ed22068f7","Type":"ContainerStarted","Data":"5ac6a5cbe81cca68176725dc3409bff75225087eb39b7e8f8f75e6370314c34d"} Aug 01 00:29:44.016463 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:44.016354 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:29:44.017664 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:44.017642 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-nlksz" event={"ID":"a878cee9-835b-4d43-9d83-b5469f709610","Type":"ContainerStarted","Data":"7d306e359dede0d9dfaa03cb1b0c7e413f7b753f24396771f30cc0da3d5e47f2"} Aug 01 00:29:44.027350 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:44.027310 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" podStartSLOduration=1.222192287 podStartE2EDuration="6.02729758s" totalImagesPullingTime="4.805105293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:44.026954763 +0000 UTC m=+72.815676585" watchObservedRunningTime="2026-08-01 00:29:44.02729758 +0000 UTC m=+72.816019406" Aug 01 00:29:44.046122 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:44.045900 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-nlksz" podStartSLOduration=1.300473368 podStartE2EDuration="6.045883215s" totalImagesPullingTime="4.745409847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:44.042930628 +0000 UTC m=+72.831652454" watchObservedRunningTime="2026-08-01 00:29:44.045883215 +0000 UTC m=+72.834605041" Aug 01 00:29:44.075354 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:44.075302 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nlw9l" podStartSLOduration=65.16620058 podStartE2EDuration="1m11.075284425s" totalImagesPullingTime="5.909083845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:44.074651796 +0000 UTC m=+72.863373623" watchObservedRunningTime="2026-08-01 00:29:44.075284425 +0000 UTC m=+72.864006252" Aug 01 00:29:44.075496 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:44.075393 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z87nn" podStartSLOduration=65.988559046 podStartE2EDuration="1m12.075384466s" totalImagesPullingTime="6.08682542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:44.05783629 +0000 UTC m=+72.846558117" watchObservedRunningTime="2026-08-01 00:29:44.075384466 +0000 UTC m=+72.864106293" Aug 01 00:29:45.876332 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:45.876302 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8x7hp"] Aug 01 00:29:45.880813 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:45.880787 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:45.883126 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:45.883104 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 01 00:29:45.883209 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:45.883122 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9ldw2\"" Aug 01 00:29:45.883209 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:45.883104 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 01 00:29:45.890457 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:45.890437 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8x7hp"] Aug 01 00:29:45.908857 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:45.908834 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:45.908956 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:45.908863 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2c9\" (UniqueName: \"kubernetes.io/projected/e52c1e00-286c-4c75-b6c1-54481a442121-kube-api-access-jl2c9\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:45.908956 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:45.908898 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e52c1e00-286c-4c75-b6c1-54481a442121-crio-socket\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:45.908956 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:45.908949 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:45.909069 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:45.909022 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e52c1e00-286c-4c75-b6c1-54481a442121-data-volume\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:46.009413 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:46.009372 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e52c1e00-286c-4c75-b6c1-54481a442121-data-volume\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:46.009578 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:46.009444 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:46.009578 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:46.009464 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2c9\" (UniqueName: \"kubernetes.io/projected/e52c1e00-286c-4c75-b6c1-54481a442121-kube-api-access-jl2c9\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:46.009578 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:46.009494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e52c1e00-286c-4c75-b6c1-54481a442121-crio-socket\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:46.009578 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:46.009513 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:46.009578 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:46.009517 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:29:46.009759 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:46.009599 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm podName:e52c1e00-286c-4c75-b6c1-54481a442121 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:46.509579034 +0000 UTC m=+75.298300839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x7hp" (UID: "e52c1e00-286c-4c75-b6c1-54481a442121") : configmap "kube-rbac-proxy" not found Aug 01 00:29:46.009759 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:46.009619 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e52c1e00-286c-4c75-b6c1-54481a442121-crio-socket\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:46.009759 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:46.009626 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:29:46.009759 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:46.009686 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls podName:e52c1e00-286c-4c75-b6c1-54481a442121 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:46.509673508 +0000 UTC m=+75.298395315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x7hp" (UID: "e52c1e00-286c-4c75-b6c1-54481a442121") : secret "insights-runtime-extractor-tls" not found Aug 01 00:29:46.009759 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:46.009709 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e52c1e00-286c-4c75-b6c1-54481a442121-data-volume\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:46.017363 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:46.017341 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2c9\" (UniqueName: \"kubernetes.io/projected/e52c1e00-286c-4c75-b6c1-54481a442121-kube-api-access-jl2c9\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:46.110865 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:46.110812 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:46.111044 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:46.110963 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 01 00:29:46.111044 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:46.111031 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls podName:802cbf7b-10e0-4ff6-8443-c0f7b60a9727 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:54.111016008 +0000 UTC m=+82.899737813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-znmsh" (UID: "802cbf7b-10e0-4ff6-8443-c0f7b60a9727") : secret "cluster-monitoring-operator-tls" not found Aug 01 00:29:46.212423 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:46.212313 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-t2tnh\" (UID: \"209f7a64-59e3-4d72-9b5a-22f72f01e2dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:29:46.212574 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:46.212468 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 01 00:29:46.212574 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:46.212531 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls podName:209f7a64-59e3-4d72-9b5a-22f72f01e2dc nodeName:}" failed. No retries permitted until 2026-08-01 00:29:54.212514862 +0000 UTC m=+83.001236666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-t2tnh" (UID: "209f7a64-59e3-4d72-9b5a-22f72f01e2dc") : secret "samples-operator-tls" not found Aug 01 00:29:46.514521 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:46.514435 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:46.514661 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:46.514527 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:46.514661 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:46.514586 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:29:46.514661 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:46.514587 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:29:46.514661 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:46.514635 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm podName:e52c1e00-286c-4c75-b6c1-54481a442121 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:47.514621335 +0000 UTC m=+76.303343140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x7hp" (UID: "e52c1e00-286c-4c75-b6c1-54481a442121") : configmap "kube-rbac-proxy" not found Aug 01 00:29:46.514661 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:46.514647 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls podName:e52c1e00-286c-4c75-b6c1-54481a442121 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:47.514641596 +0000 UTC m=+76.303363399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x7hp" (UID: "e52c1e00-286c-4c75-b6c1-54481a442121") : secret "insights-runtime-extractor-tls" not found Aug 01 00:29:47.523596 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:47.523554 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:47.523981 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:47.523612 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:47.523981 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:47.523684 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:29:47.523981 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:47.523707 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:29:47.523981 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:47.523746 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm podName:e52c1e00-286c-4c75-b6c1-54481a442121 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:49.523730277 +0000 UTC m=+78.312452086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x7hp" (UID: "e52c1e00-286c-4c75-b6c1-54481a442121") : configmap "kube-rbac-proxy" not found Aug 01 00:29:47.523981 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:47.523761 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls podName:e52c1e00-286c-4c75-b6c1-54481a442121 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:49.523754568 +0000 UTC m=+78.312476372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x7hp" (UID: "e52c1e00-286c-4c75-b6c1-54481a442121") : secret "insights-runtime-extractor-tls" not found Aug 01 00:29:48.430514 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:48.430485 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5b74f_a434193d-a324-4f56-90ae-82de780986a4/dns-node-resolver/0.log" Aug 01 00:29:49.429371 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:49.429344 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8w9d6_fee8d179-f489-4751-895d-b5294aa866fb/node-ca/0.log" Aug 01 00:29:49.539359 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:49.539319 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:49.539555 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:49.539385 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:49.539555 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:49.539443 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:29:49.539555 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:49.539506 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:29:49.539555 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:49.539512 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm podName:e52c1e00-286c-4c75-b6c1-54481a442121 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:53.539496453 +0000 UTC m=+82.328218261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x7hp" (UID: "e52c1e00-286c-4c75-b6c1-54481a442121") : configmap "kube-rbac-proxy" not found Aug 01 00:29:49.539732 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:49.539569 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls podName:e52c1e00-286c-4c75-b6c1-54481a442121 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:53.539553718 +0000 UTC m=+82.328275527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x7hp" (UID: "e52c1e00-286c-4c75-b6c1-54481a442121") : secret "insights-runtime-extractor-tls" not found Aug 01 00:29:50.433132 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:50.433091 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5kstw_1a6ef2cf-9383-4287-8b3d-5fc395998c22/serve-healthcheck-canary/0.log" Aug 01 00:29:53.575532 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:53.575498 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:53.575916 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:53.575553 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:29:53.575916 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:53.575625 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:29:53.575916 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:53.575647 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:29:53.575916 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:53.575682 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm podName:e52c1e00-286c-4c75-b6c1-54481a442121 nodeName:}" failed. No retries permitted until 2026-08-01 00:30:01.5756668 +0000 UTC m=+90.364388605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x7hp" (UID: "e52c1e00-286c-4c75-b6c1-54481a442121") : configmap "kube-rbac-proxy" not found Aug 01 00:29:53.575916 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:53.575697 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls podName:e52c1e00-286c-4c75-b6c1-54481a442121 nodeName:}" failed. No retries permitted until 2026-08-01 00:30:01.5756917 +0000 UTC m=+90.364413503 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x7hp" (UID: "e52c1e00-286c-4c75-b6c1-54481a442121") : secret "insights-runtime-extractor-tls" not found Aug 01 00:29:54.179951 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:54.179915 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:29:54.180151 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:54.180103 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 01 00:29:54.180194 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:54.180172 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls podName:802cbf7b-10e0-4ff6-8443-c0f7b60a9727 nodeName:}" failed. No retries permitted until 2026-08-01 00:30:10.180157221 +0000 UTC m=+98.968879024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-znmsh" (UID: "802cbf7b-10e0-4ff6-8443-c0f7b60a9727") : secret "cluster-monitoring-operator-tls" not found Aug 01 00:29:54.280920 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:29:54.280886 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-t2tnh\" (UID: \"209f7a64-59e3-4d72-9b5a-22f72f01e2dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:29:54.281116 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:54.281008 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 01 00:29:54.281116 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:29:54.281082 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls podName:209f7a64-59e3-4d72-9b5a-22f72f01e2dc nodeName:}" failed. No retries permitted until 2026-08-01 00:30:10.281050457 +0000 UTC m=+99.069772262 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-t2tnh" (UID: "209f7a64-59e3-4d72-9b5a-22f72f01e2dc") : secret "samples-operator-tls" not found Aug 01 00:30:01.639009 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:01.638980 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:30:01.639416 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:01.639036 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:30:01.639416 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:30:01.639140 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:30:01.639416 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:30:01.639205 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls podName:e52c1e00-286c-4c75-b6c1-54481a442121 nodeName:}" failed. No retries permitted until 2026-08-01 00:30:17.639189137 +0000 UTC m=+106.427910960 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x7hp" (UID: "e52c1e00-286c-4c75-b6c1-54481a442121") : secret "insights-runtime-extractor-tls" not found Aug 01 00:30:01.639525 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:01.639468 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e52c1e00-286c-4c75-b6c1-54481a442121-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:30:08.562353 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.562320 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk"] Aug 01 00:30:08.566541 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.566524 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk" Aug 01 00:30:08.568925 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.568905 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 01 00:30:08.568925 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.568910 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-sbfbp\"" Aug 01 00:30:08.570280 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.570263 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 01 00:30:08.570374 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.570300 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 01 00:30:08.570473 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.570461 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Aug 01 00:30:08.574434 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.574415 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk"] Aug 01 00:30:08.693663 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.693630 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk\" (UID: \"fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk" Aug 01 00:30:08.693827 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.693748 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wkjn\" (UniqueName: \"kubernetes.io/projected/fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0-kube-api-access-4wkjn\") pod \"managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk\" (UID: \"fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk" Aug 01 00:30:08.794452 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.794418 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk\" (UID: \"fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk" Aug 01 00:30:08.794667 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.794647 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkjn\" (UniqueName: \"kubernetes.io/projected/fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0-kube-api-access-4wkjn\") pod \"managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk\" (UID: \"fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk" Aug 01 00:30:08.796817 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.796794 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk\" (UID: \"fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk" Aug 01 00:30:08.803673 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.803653 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkjn\" (UniqueName: \"kubernetes.io/projected/fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0-kube-api-access-4wkjn\") pod \"managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk\" (UID: \"fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk" Aug 01 00:30:08.886080 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:08.885997 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk" Aug 01 00:30:09.006153 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:09.006130 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk"] Aug 01 00:30:09.008346 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:30:09.008320 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfec7c4b0_f0da_48ef_8a7a_8e3fffaa50a0.slice/crio-c2d5fe2b234d2fbbbcd6fd8954509e9825d846115311e60ed16e8c645c6535d8 WatchSource:0}: Error finding container c2d5fe2b234d2fbbbcd6fd8954509e9825d846115311e60ed16e8c645c6535d8: Status 404 returned error can't find the container with id c2d5fe2b234d2fbbbcd6fd8954509e9825d846115311e60ed16e8c645c6535d8 Aug 01 00:30:09.086118 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:09.086050 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk" event={"ID":"fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0","Type":"ContainerStarted","Data":"c2d5fe2b234d2fbbbcd6fd8954509e9825d846115311e60ed16e8c645c6535d8"} Aug 01 00:30:09.500334 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:09.500292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:30:09.500538 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:09.500408 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:30:09.502604 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:09.502579 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5b71018-3dd9-43cc-88e1-220d978adcf6-metrics-tls\") pod \"dns-default-d7hj6\" (UID: \"e5b71018-3dd9-43cc-88e1-220d978adcf6\") " pod="openshift-dns/dns-default-d7hj6" Aug 01 00:30:09.502714 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:09.502681 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls\") pod \"image-registry-6fcd8946f6-l8kx8\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:30:09.523031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:09.523005 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fl89b\"" Aug 01 00:30:09.531158 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:09.531144 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7hj6" Aug 01 00:30:09.651824 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:09.651798 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7hj6"] Aug 01 00:30:09.654042 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:30:09.654018 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b71018_3dd9_43cc_88e1_220d978adcf6.slice/crio-82deac4f51acc4721445fcde6b663cc3492b5fcdf1ff33c65e14357efe7e3eb9 WatchSource:0}: Error finding container 82deac4f51acc4721445fcde6b663cc3492b5fcdf1ff33c65e14357efe7e3eb9: Status 404 returned error can't find the container with id 82deac4f51acc4721445fcde6b663cc3492b5fcdf1ff33c65e14357efe7e3eb9 Aug 01 00:30:09.796126 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:09.796040 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4jrsr\"" Aug 01 00:30:09.804229 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:09.804202 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:30:09.923919 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:09.923895 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fcd8946f6-l8kx8"] Aug 01 00:30:09.926209 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:30:09.926183 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dc1c37_a456_45c8_a01c_c30082421e3b.slice/crio-619c87143c32c9526fafa7c4e485d51bb3b88f65eb40d80c0334fad22e09c0d2 WatchSource:0}: Error finding container 619c87143c32c9526fafa7c4e485d51bb3b88f65eb40d80c0334fad22e09c0d2: Status 404 returned error can't find the container with id 619c87143c32c9526fafa7c4e485d51bb3b88f65eb40d80c0334fad22e09c0d2 Aug 01 00:30:10.098452 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:10.098406 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7hj6" event={"ID":"e5b71018-3dd9-43cc-88e1-220d978adcf6","Type":"ContainerStarted","Data":"82deac4f51acc4721445fcde6b663cc3492b5fcdf1ff33c65e14357efe7e3eb9"} Aug 01 00:30:10.101097 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:10.101051 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" event={"ID":"e5dc1c37-a456-45c8-a01c-c30082421e3b","Type":"ContainerStarted","Data":"f3e6faf5381eecec99dcb2352ea7af1598cd72a19c146c2865228237db5bcc58"} Aug 01 00:30:10.101245 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:10.101115 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" event={"ID":"e5dc1c37-a456-45c8-a01c-c30082421e3b","Type":"ContainerStarted","Data":"619c87143c32c9526fafa7c4e485d51bb3b88f65eb40d80c0334fad22e09c0d2"} Aug 01 00:30:10.101309 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:10.101265 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:30:10.134434 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:10.134379 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" podStartSLOduration=97.13436007 podStartE2EDuration="1m37.13436007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:10.133955443 +0000 UTC m=+98.922677270" watchObservedRunningTime="2026-08-01 00:30:10.13436007 +0000 UTC m=+98.923081898" Aug 01 00:30:10.206530 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:10.206489 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:30:10.213393 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:10.213356 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/802cbf7b-10e0-4ff6-8443-c0f7b60a9727-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-znmsh\" (UID: \"802cbf7b-10e0-4ff6-8443-c0f7b60a9727\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:30:10.223120 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:10.223087 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" Aug 01 00:30:10.309113 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:10.308694 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-t2tnh\" (UID: \"209f7a64-59e3-4d72-9b5a-22f72f01e2dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:30:10.314406 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:10.314279 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/209f7a64-59e3-4d72-9b5a-22f72f01e2dc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-t2tnh\" (UID: \"209f7a64-59e3-4d72-9b5a-22f72f01e2dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:30:10.332014 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:10.331982 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" Aug 01 00:30:10.401988 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:10.401862 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh"] Aug 01 00:30:10.502178 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:10.502091 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh"] Aug 01 00:30:11.106714 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.106671 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" event={"ID":"209f7a64-59e3-4d72-9b5a-22f72f01e2dc","Type":"ContainerStarted","Data":"24db1798d8869c11db6968b67824660daf52bffdc5bfee619547c571f4c95abe"} Aug 01 00:30:11.108271 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.108239 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" event={"ID":"802cbf7b-10e0-4ff6-8443-c0f7b60a9727","Type":"ContainerStarted","Data":"e3fad24828c24cf60fec9ca5b0771edb38e64be9d0d5ad3dd5e975a6cbfbae6d"} Aug 01 00:30:11.385767 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.385736 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-546bd95747-cvtg4"] Aug 01 00:30:11.390226 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.390203 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.394103 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.394082 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 01 00:30:11.394378 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.394357 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 01 00:30:11.394475 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.394360 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mc5mx\"" Aug 01 00:30:11.394475 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.394423 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 01 00:30:11.394475 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.394443 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 01 00:30:11.394475 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.394468 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 01 00:30:11.394676 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.394426 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 01 00:30:11.394676 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.394602 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 01 00:30:11.403732 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.403712 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546bd95747-cvtg4"] Aug 01 00:30:11.521477 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.521441 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-oauth-config\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.521625 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.521522 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-oauth-serving-cert\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.521625 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.521554 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-config\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.521625 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.521594 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-serving-cert\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.521720 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.521623 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb96t\" (UniqueName: \"kubernetes.io/projected/c9ad886e-3b36-48fc-806e-2c291e9a553e-kube-api-access-zb96t\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.521720 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.521697 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-service-ca\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.622922 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.622886 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-serving-cert\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.623115 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.622929 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb96t\" (UniqueName: \"kubernetes.io/projected/c9ad886e-3b36-48fc-806e-2c291e9a553e-kube-api-access-zb96t\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.623115 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.623083 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-service-ca\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.623242 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.623215 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-oauth-config\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.623300 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.623257 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-oauth-serving-cert\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.623300 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.623278 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-config\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.623943 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.623895 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-config\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.623943 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.623925 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-service-ca\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.624118 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.623976 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-oauth-serving-cert\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.625573 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.625546 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-serving-cert\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.625651 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.625610 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-oauth-config\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.632586 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.632543 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb96t\" (UniqueName: \"kubernetes.io/projected/c9ad886e-3b36-48fc-806e-2c291e9a553e-kube-api-access-zb96t\") pod \"console-546bd95747-cvtg4\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:11.701765 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:11.701685 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:12.664414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:12.663928 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546bd95747-cvtg4"] Aug 01 00:30:12.669248 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:30:12.669187 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ad886e_3b36_48fc_806e_2c291e9a553e.slice/crio-1a578a6ef956bb4d3e81a3c1bf0579f85196e0d76471e85ea70203d3781eae9c WatchSource:0}: Error finding container 1a578a6ef956bb4d3e81a3c1bf0579f85196e0d76471e85ea70203d3781eae9c: Status 404 returned error can't find the container with id 1a578a6ef956bb4d3e81a3c1bf0579f85196e0d76471e85ea70203d3781eae9c Aug 01 00:30:13.121519 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:13.121451 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk" event={"ID":"fec7c4b0-f0da-48ef-8a7a-8e3fffaa50a0","Type":"ContainerStarted","Data":"f668fa2ecf0df51db88addae3ceb61374c4785dca1ffe9103a0827ce6265f0b2"} Aug 01 00:30:13.123574 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:13.123546 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7hj6" event={"ID":"e5b71018-3dd9-43cc-88e1-220d978adcf6","Type":"ContainerStarted","Data":"f2d75a9d7f445cf2b0c03a7ca8834cc01c3ca89eae25fc3b6d175a270f411955"} Aug 01 00:30:13.123693 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:13.123614 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7hj6" event={"ID":"e5b71018-3dd9-43cc-88e1-220d978adcf6","Type":"ContainerStarted","Data":"0ff65807c30ea29b54b019ff836924a547b749e53391e10857579acd5896d019"} Aug 01 00:30:13.123693 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:13.123644 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d7hj6" Aug 01 00:30:13.124957 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:13.124916 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546bd95747-cvtg4" event={"ID":"c9ad886e-3b36-48fc-806e-2c291e9a553e","Type":"ContainerStarted","Data":"1a578a6ef956bb4d3e81a3c1bf0579f85196e0d76471e85ea70203d3781eae9c"} Aug 01 00:30:13.135184 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:13.135146 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b8574dc5d-zz8gk" podStartSLOduration=1.635492577 podStartE2EDuration="5.13513375s" totalImagesPullingTime="3.499641173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:13.135026334 +0000 UTC m=+101.923748161" watchObservedRunningTime="2026-08-01 00:30:13.13513375 +0000 UTC m=+101.923855567" Aug 01 00:30:13.153201 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:13.153148 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d7hj6" podStartSLOduration=65.298279578 podStartE2EDuration="1m8.153130301s" totalImagesPullingTime="2.854850723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:13.152099006 +0000 UTC m=+101.940820831" watchObservedRunningTime="2026-08-01 00:30:13.153130301 +0000 UTC m=+101.941852174" Aug 01 00:30:15.024149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:15.024116 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z87nn" Aug 01 00:30:15.133677 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:15.133641 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" event={"ID":"209f7a64-59e3-4d72-9b5a-22f72f01e2dc","Type":"ContainerStarted","Data":"f435ee168fa89fe2bc302da5082b44520ca836d12e6af0b2819acb26c5532b3b"} Aug 01 00:30:15.133851 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:15.133705 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" event={"ID":"209f7a64-59e3-4d72-9b5a-22f72f01e2dc","Type":"ContainerStarted","Data":"7ac11c361dfa5cb1baa34b40a3c822f0fb49ac31c0191fc573bc1ca7a6c19c84"} Aug 01 00:30:15.135119 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:15.135095 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" event={"ID":"802cbf7b-10e0-4ff6-8443-c0f7b60a9727","Type":"ContainerStarted","Data":"619272bd3741164b38da55f0485150de5de6dedea5629fcc5a7f13eb982fd006"} Aug 01 00:30:15.149507 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:15.149455 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-t2tnh" podStartSLOduration=33.469698019 podStartE2EDuration="37.149440548s" totalImagesPullingTime="3.679742529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:15.148522774 +0000 UTC m=+103.937244600" watchObservedRunningTime="2026-08-01 00:30:15.149440548 +0000 UTC m=+103.938162373" Aug 01 00:30:15.164401 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:15.164352 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" podStartSLOduration=33.342870007 podStartE2EDuration="37.164333494s" totalImagesPullingTime="3.821463487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:15.163266861 +0000 UTC m=+103.951988687" watchObservedRunningTime="2026-08-01 00:30:15.164333494 +0000 UTC m=+103.953055321" Aug 01 00:30:17.143226 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:17.143185 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546bd95747-cvtg4" event={"ID":"c9ad886e-3b36-48fc-806e-2c291e9a553e","Type":"ContainerStarted","Data":"e6c7ee73db64667ba14a8c1a9eefe046dbfd08411e83a804b455b6063401ac2d"} Aug 01 00:30:17.163896 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:17.163847 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-546bd95747-cvtg4" podStartSLOduration=2.250228468 podStartE2EDuration="6.163832399s" totalImagesPullingTime="3.913603931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:17.162459522 +0000 UTC m=+105.951181367" watchObservedRunningTime="2026-08-01 00:30:17.163832399 +0000 UTC m=+105.952554222" Aug 01 00:30:17.678787 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:17.678744 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:30:17.681041 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:17.681010 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e52c1e00-286c-4c75-b6c1-54481a442121-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x7hp\" (UID: \"e52c1e00-286c-4c75-b6c1-54481a442121\") " pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:30:17.690007 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:17.689984 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8x7hp" Aug 01 00:30:17.835356 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:17.835332 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8x7hp"] Aug 01 00:30:17.837668 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:30:17.837638 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52c1e00_286c_4c75_b6c1_54481a442121.slice/crio-09d4638da64cdd11d238588c44f20c7b20554b6eff24a93e1f28c7bedef108fc WatchSource:0}: Error finding container 09d4638da64cdd11d238588c44f20c7b20554b6eff24a93e1f28c7bedef108fc: Status 404 returned error can't find the container with id 09d4638da64cdd11d238588c44f20c7b20554b6eff24a93e1f28c7bedef108fc Aug 01 00:30:18.148114 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:18.148050 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8x7hp" event={"ID":"e52c1e00-286c-4c75-b6c1-54481a442121","Type":"ContainerStarted","Data":"26a7639e2853325e685b29a006c5960f5e6ac069180788889fed73dbfa1c4148"} Aug 01 00:30:18.148470 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:18.148140 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8x7hp" event={"ID":"e52c1e00-286c-4c75-b6c1-54481a442121","Type":"ContainerStarted","Data":"09d4638da64cdd11d238588c44f20c7b20554b6eff24a93e1f28c7bedef108fc"} Aug 01 00:30:19.153395 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:19.153356 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8x7hp" event={"ID":"e52c1e00-286c-4c75-b6c1-54481a442121","Type":"ContainerStarted","Data":"9ecf3c555160b8994a7304df309ce500455fa36569e0c79fcdacf66669950e82"} Aug 01 00:30:20.158053 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:20.157965 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8x7hp" event={"ID":"e52c1e00-286c-4c75-b6c1-54481a442121","Type":"ContainerStarted","Data":"3756c8edcae506ca90c506533ad3d5d996f1167a5b1c0f50ee07e4abcf976305"} Aug 01 00:30:20.175243 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:20.175192 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8x7hp" podStartSLOduration=33.30051996 podStartE2EDuration="35.175176444s" totalImagesPullingTime="1.874656484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:29:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:20.17434048 +0000 UTC m=+108.963062315" watchObservedRunningTime="2026-08-01 00:30:20.175176444 +0000 UTC m=+108.963898269" Aug 01 00:30:21.701894 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:21.701864 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:21.702312 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:21.701934 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:21.707170 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:21.707137 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:22.168401 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:22.168372 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:23.130642 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.130612 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d7hj6" Aug 01 00:30:23.264475 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.264437 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms"] Aug 01 00:30:23.267720 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.267666 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.270955 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.270923 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Aug 01 00:30:23.271113 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.271018 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wjttd\"" Aug 01 00:30:23.271263 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.271247 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Aug 01 00:30:23.271312 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.271277 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 01 00:30:23.271363 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.271247 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Aug 01 00:30:23.277417 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.277395 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mvkhk"] Aug 01 00:30:23.280456 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.280434 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms"] Aug 01 00:30:23.280581 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.280562 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.283050 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.283028 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x7kb2\"" Aug 01 00:30:23.283185 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.283164 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 01 00:30:23.283260 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.283196 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 01 00:30:23.283415 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.283400 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 01 00:30:23.427347 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427253 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33ed01ee-85e2-444b-b942-93f4c2393b35-sys\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.427347 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427298 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33ed01ee-85e2-444b-b942-93f4c2393b35-root\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.427534 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427355 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-accelerators-collector-config\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.427534 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427397 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6b43079d-4e13-4651-98a3-47a97f87fd82-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.427534 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427417 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.427534 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427447 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhnb5\" (UniqueName: \"kubernetes.io/projected/33ed01ee-85e2-444b-b942-93f4c2393b35-kube-api-access-vhnb5\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.427534 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427516 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.427710 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427557 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33ed01ee-85e2-444b-b942-93f4c2393b35-metrics-client-ca\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.427710 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427578 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-wtmp\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.427710 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427596 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b43079d-4e13-4651-98a3-47a97f87fd82-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.427710 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427615 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-textfile\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.427710 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427642 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.427710 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427685 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.427710 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427710 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-tls\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.427956 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.427745 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96fdw\" (UniqueName: \"kubernetes.io/projected/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-api-access-96fdw\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.528911 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.528869 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.529097 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.528919 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33ed01ee-85e2-444b-b942-93f4c2393b35-metrics-client-ca\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529097 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.528942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-wtmp\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529097 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529047 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b43079d-4e13-4651-98a3-47a97f87fd82-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.529097 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529092 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-wtmp\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529100 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-textfile\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529161 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.529414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.529414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529232 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-tls\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96fdw\" (UniqueName: \"kubernetes.io/projected/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-api-access-96fdw\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.529414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529317 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33ed01ee-85e2-444b-b942-93f4c2393b35-sys\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529353 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33ed01ee-85e2-444b-b942-93f4c2393b35-root\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529381 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-accelerators-collector-config\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529414 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529399 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-textfile\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529952 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529405 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6b43079d-4e13-4651-98a3-47a97f87fd82-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.529952 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529457 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529952 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529501 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhnb5\" (UniqueName: \"kubernetes.io/projected/33ed01ee-85e2-444b-b942-93f4c2393b35-kube-api-access-vhnb5\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529952 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529622 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33ed01ee-85e2-444b-b942-93f4c2393b35-metrics-client-ca\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529952 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529694 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33ed01ee-85e2-444b-b942-93f4c2393b35-sys\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529952 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529704 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6b43079d-4e13-4651-98a3-47a97f87fd82-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.529952 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:30:23.529744 2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Aug 01 00:30:23.529952 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529765 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33ed01ee-85e2-444b-b942-93f4c2393b35-root\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.529952 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529796 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b43079d-4e13-4651-98a3-47a97f87fd82-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.529952 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:30:23.529814 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-state-metrics-tls podName:6b43079d-4e13-4651-98a3-47a97f87fd82 nodeName:}" failed. No retries permitted until 2026-08-01 00:30:24.029797622 +0000 UTC m=+112.818519445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-state-metrics-tls") pod "kube-state-metrics-6bd9664c4-h9tms" (UID: "6b43079d-4e13-4651-98a3-47a97f87fd82") : secret "kube-state-metrics-tls" not found Aug 01 00:30:23.529952 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.529816 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.530512 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.530194 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-accelerators-collector-config\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.531926 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.531897 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-tls\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.531926 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.531911 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.532242 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.532223 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33ed01ee-85e2-444b-b942-93f4c2393b35-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.540352 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.540325 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhnb5\" (UniqueName: \"kubernetes.io/projected/33ed01ee-85e2-444b-b942-93f4c2393b35-kube-api-access-vhnb5\") pod \"node-exporter-mvkhk\" (UID: \"33ed01ee-85e2-444b-b942-93f4c2393b35\") " pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.540482 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.540462 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96fdw\" (UniqueName: \"kubernetes.io/projected/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-api-access-96fdw\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:23.589897 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:23.589852 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mvkhk" Aug 01 00:30:23.601577 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:30:23.601540 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ed01ee_85e2_444b_b942_93f4c2393b35.slice/crio-d467d131e4685452def06011d54073a6a7b654f5f3aa32ef71173aba04b99488 WatchSource:0}: Error finding container d467d131e4685452def06011d54073a6a7b654f5f3aa32ef71173aba04b99488: Status 404 returned error can't find the container with id d467d131e4685452def06011d54073a6a7b654f5f3aa32ef71173aba04b99488 Aug 01 00:30:24.033417 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:24.033381 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:24.035835 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:24.035792 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b43079d-4e13-4651-98a3-47a97f87fd82-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-h9tms\" (UID: \"6b43079d-4e13-4651-98a3-47a97f87fd82\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:24.172829 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:24.172787 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvkhk" event={"ID":"33ed01ee-85e2-444b-b942-93f4c2393b35","Type":"ContainerStarted","Data":"d467d131e4685452def06011d54073a6a7b654f5f3aa32ef71173aba04b99488"} Aug 01 00:30:24.178182 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:24.178151 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" Aug 01 00:30:24.338854 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:24.338825 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms"] Aug 01 00:30:24.340080 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:30:24.340037 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b43079d_4e13_4651_98a3_47a97f87fd82.slice/crio-e88f9c8a9e31b65c311e09ee92edf85189ce473a8018682a32351b2a310a07c1 WatchSource:0}: Error finding container e88f9c8a9e31b65c311e09ee92edf85189ce473a8018682a32351b2a310a07c1: Status 404 returned error can't find the container with id e88f9c8a9e31b65c311e09ee92edf85189ce473a8018682a32351b2a310a07c1 Aug 01 00:30:25.177499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.177406 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" event={"ID":"6b43079d-4e13-4651-98a3-47a97f87fd82","Type":"ContainerStarted","Data":"e88f9c8a9e31b65c311e09ee92edf85189ce473a8018682a32351b2a310a07c1"} Aug 01 00:30:25.178953 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.178928 2563 generic.go:356] "Generic (PLEG): container finished" podID="33ed01ee-85e2-444b-b942-93f4c2393b35" containerID="9ba3a97947253d5ccd7ad5bba35181f564b2feb6067d512b032504bf6d23b112" exitCode=0 Aug 01 00:30:25.179091 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.178981 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvkhk" event={"ID":"33ed01ee-85e2-444b-b942-93f4c2393b35","Type":"ContainerDied","Data":"9ba3a97947253d5ccd7ad5bba35181f564b2feb6067d512b032504bf6d23b112"} Aug 01 00:30:25.226179 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.226151 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-75755945d6-p6b8l"] Aug 01 00:30:25.234426 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.234399 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.236697 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.236666 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Aug 01 00:30:25.236833 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.236698 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Aug 01 00:30:25.236833 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.236748 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Aug 01 00:30:25.236833 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.236821 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Aug 01 00:30:25.237143 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.237048 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Aug 01 00:30:25.237229 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.237072 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3j45mfbh1ljaa\"" Aug 01 00:30:25.237229 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.237192 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9l5qt\"" Aug 01 00:30:25.241338 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.241317 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75755945d6-p6b8l"] Aug 01 00:30:25.344889 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.344849 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.344889 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.344890 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhk57\" (UniqueName: \"kubernetes.io/projected/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-kube-api-access-xhk57\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.345134 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.344909 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.345134 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.344954 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-grpc-tls\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.345134 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.345040 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.345274 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.345139 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.345274 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.345175 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-tls\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.345274 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.345198 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-metrics-client-ca\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.446302 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.446211 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-grpc-tls\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.446302 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.446283 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.446496 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.446324 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.446496 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.446356 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-tls\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.446496 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.446384 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-metrics-client-ca\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.446496 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.446440 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.446496 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.446462 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhk57\" (UniqueName: \"kubernetes.io/projected/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-kube-api-access-xhk57\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.446496 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.446487 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.447275 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.447239 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-metrics-client-ca\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.448963 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.448939 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.449052 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.449001 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.449257 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.449239 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-grpc-tls\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.449292 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.449266 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.449452 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.449430 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.449491 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.449455 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-secret-thanos-querier-tls\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.454368 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.454340 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhk57\" (UniqueName: \"kubernetes.io/projected/3f3349b2-3e23-48e3-89d8-76e3fdb6d354-kube-api-access-xhk57\") pod \"thanos-querier-75755945d6-p6b8l\" (UID: \"3f3349b2-3e23-48e3-89d8-76e3fdb6d354\") " pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.544841 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.544808 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:25.678383 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:25.678350 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75755945d6-p6b8l"] Aug 01 00:30:25.682360 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:30:25.682334 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3349b2_3e23_48e3_89d8_76e3fdb6d354.slice/crio-0bffd77616cf8a09d4e259ee5fbd63391ec8b0c8a060a460234ff021e1793042 WatchSource:0}: Error finding container 0bffd77616cf8a09d4e259ee5fbd63391ec8b0c8a060a460234ff021e1793042: Status 404 returned error can't find the container with id 0bffd77616cf8a09d4e259ee5fbd63391ec8b0c8a060a460234ff021e1793042 Aug 01 00:30:26.184928 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:26.184886 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvkhk" event={"ID":"33ed01ee-85e2-444b-b942-93f4c2393b35","Type":"ContainerStarted","Data":"36ad0cb1f5dded3c0812fa5cba1d45aa18b025ca6acbf0b33d08975b2f840f40"} Aug 01 00:30:26.185445 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:26.184958 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvkhk" event={"ID":"33ed01ee-85e2-444b-b942-93f4c2393b35","Type":"ContainerStarted","Data":"bd7d1d66788ebd01a5d27f2ab2fd4dfee47593afd825af90cdf81c9a5d9da513"} Aug 01 00:30:26.186110 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:26.186078 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" event={"ID":"3f3349b2-3e23-48e3-89d8-76e3fdb6d354","Type":"ContainerStarted","Data":"0bffd77616cf8a09d4e259ee5fbd63391ec8b0c8a060a460234ff021e1793042"} Aug 01 00:30:26.205413 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:26.205358 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mvkhk" podStartSLOduration=1.877579641 podStartE2EDuration="3.205341689s" totalImagesPullingTime="1.264198254s" totalInitContainerRuntime="63.563794ms" isStatefulPod=true podCreationTimestamp="2026-08-01 00:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:26.204993341 +0000 UTC m=+114.993715171" watchObservedRunningTime="2026-08-01 00:30:26.205341689 +0000 UTC m=+114.994063515" Aug 01 00:30:26.454467 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:26.454380 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546bd95747-cvtg4"] Aug 01 00:30:27.193929 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.193888 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" event={"ID":"6b43079d-4e13-4651-98a3-47a97f87fd82","Type":"ContainerStarted","Data":"161fa58e8ddf2cc5c816b20e0fb09c4331da428d7a10d26d6fa966c55694ce47"} Aug 01 00:30:27.194423 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.193961 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" event={"ID":"6b43079d-4e13-4651-98a3-47a97f87fd82","Type":"ContainerStarted","Data":"81eac3ae834ec12f4fd9f97503b743ac1c562ec358116e1b5c741219ab4dffef"} Aug 01 00:30:27.194423 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.193993 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" event={"ID":"6b43079d-4e13-4651-98a3-47a97f87fd82","Type":"ContainerStarted","Data":"17cbf444dd2b3455c9b94efb73424c573199bf99eb17bc4855606546a1857c74"} Aug 01 00:30:27.212426 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.212361 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h9tms" podStartSLOduration=1.7959477019999999 podStartE2EDuration="4.212341221s" totalImagesPullingTime="2.416393519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:27.211960155 +0000 UTC m=+116.000681981" watchObservedRunningTime="2026-08-01 00:30:27.212341221 +0000 UTC m=+116.001063048" Aug 01 00:30:27.640198 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.640155 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-65d6c48b96-kr2xn"] Aug 01 00:30:27.643831 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.643806 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.646224 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.646203 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pkmkn\"" Aug 01 00:30:27.647292 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.647268 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Aug 01 00:30:27.647418 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.647293 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7uko52enndal2\"" Aug 01 00:30:27.647418 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.647275 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 01 00:30:27.647418 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.647314 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Aug 01 00:30:27.647418 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.647268 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Aug 01 00:30:27.652528 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.652504 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65d6c48b96-kr2xn"] Aug 01 00:30:27.766320 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.766280 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/01a2397b-9138-42f4-b397-4eb2705829d3-secret-metrics-server-client-certs\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.766517 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.766337 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/01a2397b-9138-42f4-b397-4eb2705829d3-client-ca-bundle\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.766517 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.766390 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01a2397b-9138-42f4-b397-4eb2705829d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.766517 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.766423 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/01a2397b-9138-42f4-b397-4eb2705829d3-metrics-server-audit-profiles\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.766517 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.766490 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/01a2397b-9138-42f4-b397-4eb2705829d3-secret-metrics-server-tls\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.766712 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.766522 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/01a2397b-9138-42f4-b397-4eb2705829d3-audit-log\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.766712 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.766598 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp8cl\" (UniqueName: \"kubernetes.io/projected/01a2397b-9138-42f4-b397-4eb2705829d3-kube-api-access-fp8cl\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.867729 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.867689 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp8cl\" (UniqueName: \"kubernetes.io/projected/01a2397b-9138-42f4-b397-4eb2705829d3-kube-api-access-fp8cl\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.867895 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.867770 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/01a2397b-9138-42f4-b397-4eb2705829d3-secret-metrics-server-client-certs\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.867895 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.867802 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/01a2397b-9138-42f4-b397-4eb2705829d3-client-ca-bundle\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.867895 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.867839 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01a2397b-9138-42f4-b397-4eb2705829d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.867895 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.867867 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/01a2397b-9138-42f4-b397-4eb2705829d3-metrics-server-audit-profiles\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.868135 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.867931 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/01a2397b-9138-42f4-b397-4eb2705829d3-secret-metrics-server-tls\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.868135 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.867968 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/01a2397b-9138-42f4-b397-4eb2705829d3-audit-log\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.868412 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.868385 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/01a2397b-9138-42f4-b397-4eb2705829d3-audit-log\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.868739 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.868684 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01a2397b-9138-42f4-b397-4eb2705829d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.869076 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.869023 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/01a2397b-9138-42f4-b397-4eb2705829d3-metrics-server-audit-profiles\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.870947 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.870744 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/01a2397b-9138-42f4-b397-4eb2705829d3-secret-metrics-server-client-certs\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.870947 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.870845 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/01a2397b-9138-42f4-b397-4eb2705829d3-secret-metrics-server-tls\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.870947 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.870905 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/01a2397b-9138-42f4-b397-4eb2705829d3-client-ca-bundle\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.875541 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.875516 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp8cl\" (UniqueName: \"kubernetes.io/projected/01a2397b-9138-42f4-b397-4eb2705829d3-kube-api-access-fp8cl\") pod \"metrics-server-65d6c48b96-kr2xn\" (UID: \"01a2397b-9138-42f4-b397-4eb2705829d3\") " pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:27.955522 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:27.955437 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:28.015168 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.015133 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj"] Aug 01 00:30:28.019516 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.019495 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj" Aug 01 00:30:28.021903 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.021879 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Aug 01 00:30:28.022021 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.021932 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k97rp\"" Aug 01 00:30:28.027366 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.027144 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj"] Aug 01 00:30:28.118297 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.118273 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65d6c48b96-kr2xn"] Aug 01 00:30:28.120809 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:30:28.120785 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a2397b_9138_42f4_b397_4eb2705829d3.slice/crio-ca6c5ef7e3a55f4561640c0cd5a931518801a5f3b478e735a943495e0f236f00 WatchSource:0}: Error finding container ca6c5ef7e3a55f4561640c0cd5a931518801a5f3b478e735a943495e0f236f00: Status 404 returned error can't find the container with id ca6c5ef7e3a55f4561640c0cd5a931518801a5f3b478e735a943495e0f236f00 Aug 01 00:30:28.171117 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.171091 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd7c88f4-a5b2-45bd-9a87-78a902daae7c-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-vsshj\" (UID: \"dd7c88f4-a5b2-45bd-9a87-78a902daae7c\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj" Aug 01 00:30:28.171230 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.171145 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvw8l\" (UniqueName: \"kubernetes.io/projected/dd7c88f4-a5b2-45bd-9a87-78a902daae7c-kube-api-access-kvw8l\") pod \"monitoring-plugin-6dffbbd67-vsshj\" (UID: \"dd7c88f4-a5b2-45bd-9a87-78a902daae7c\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj" Aug 01 00:30:28.200222 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.200192 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" event={"ID":"3f3349b2-3e23-48e3-89d8-76e3fdb6d354","Type":"ContainerStarted","Data":"41d7e7a11718cf630f014ef0994cbcd2619841f361fdc0a72ce09d436022190f"} Aug 01 00:30:28.200592 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.200273 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" event={"ID":"3f3349b2-3e23-48e3-89d8-76e3fdb6d354","Type":"ContainerStarted","Data":"86efa86b66a5fb6220a9e61b1f0e95f41b09096a7e568b5b515af067bb692938"} Aug 01 00:30:28.200592 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.200334 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" event={"ID":"3f3349b2-3e23-48e3-89d8-76e3fdb6d354","Type":"ContainerStarted","Data":"fb0690fa4c223ad07feb29c3cde3b6a235a03363a992c41129dc9f4f730ad6e8"} Aug 01 00:30:28.201552 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.201526 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" event={"ID":"01a2397b-9138-42f4-b397-4eb2705829d3","Type":"ContainerStarted","Data":"ca6c5ef7e3a55f4561640c0cd5a931518801a5f3b478e735a943495e0f236f00"} Aug 01 00:30:28.272538 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.272433 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd7c88f4-a5b2-45bd-9a87-78a902daae7c-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-vsshj\" (UID: \"dd7c88f4-a5b2-45bd-9a87-78a902daae7c\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj" Aug 01 00:30:28.272538 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.272489 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvw8l\" (UniqueName: \"kubernetes.io/projected/dd7c88f4-a5b2-45bd-9a87-78a902daae7c-kube-api-access-kvw8l\") pod \"monitoring-plugin-6dffbbd67-vsshj\" (UID: \"dd7c88f4-a5b2-45bd-9a87-78a902daae7c\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj" Aug 01 00:30:28.274907 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.274873 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd7c88f4-a5b2-45bd-9a87-78a902daae7c-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-vsshj\" (UID: \"dd7c88f4-a5b2-45bd-9a87-78a902daae7c\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj" Aug 01 00:30:28.280183 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.280159 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvw8l\" (UniqueName: \"kubernetes.io/projected/dd7c88f4-a5b2-45bd-9a87-78a902daae7c-kube-api-access-kvw8l\") pod \"monitoring-plugin-6dffbbd67-vsshj\" (UID: \"dd7c88f4-a5b2-45bd-9a87-78a902daae7c\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj" Aug 01 00:30:28.343985 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.343944 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj" Aug 01 00:30:28.482744 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:28.482718 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj"] Aug 01 00:30:28.485284 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:30:28.485253 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7c88f4_a5b2_45bd_9a87_78a902daae7c.slice/crio-975690a38771eb1e790d06ede0fa9d0e424e39131ee925e7a776450878e2e3bf WatchSource:0}: Error finding container 975690a38771eb1e790d06ede0fa9d0e424e39131ee925e7a776450878e2e3bf: Status 404 returned error can't find the container with id 975690a38771eb1e790d06ede0fa9d0e424e39131ee925e7a776450878e2e3bf Aug 01 00:30:29.207301 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.207262 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj" event={"ID":"dd7c88f4-a5b2-45bd-9a87-78a902daae7c","Type":"ContainerStarted","Data":"975690a38771eb1e790d06ede0fa9d0e424e39131ee925e7a776450878e2e3bf"} Aug 01 00:30:29.513373 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.513344 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 01 00:30:29.517325 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.517302 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.519964 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.519937 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qsgq6\"" Aug 01 00:30:29.520089 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.519961 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Aug 01 00:30:29.520089 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.520032 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Aug 01 00:30:29.520089 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.520078 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Aug 01 00:30:29.520258 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.520139 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Aug 01 00:30:29.520433 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.520366 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Aug 01 00:30:29.520433 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.520370 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Aug 01 00:30:29.521107 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.521085 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Aug 01 00:30:29.521215 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.521129 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Aug 01 00:30:29.521306 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.521287 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Aug 01 00:30:29.521539 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.521511 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Aug 01 00:30:29.521634 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.521544 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9sdooqidf6i5e\"" Aug 01 00:30:29.521634 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.521552 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Aug 01 00:30:29.521634 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.521581 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Aug 01 00:30:29.524233 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.524205 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Aug 01 00:30:29.527204 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.527183 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Aug 01 00:30:29.534770 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.534748 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 01 00:30:29.585561 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.585518 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.585561 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.585564 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.585764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.585590 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.585764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.585643 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/48998805-0245-4641-a435-23d88664daca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.585764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.585685 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cfb7\" (UniqueName: \"kubernetes.io/projected/48998805-0245-4641-a435-23d88664daca-kube-api-access-8cfb7\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.585764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.585739 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48998805-0245-4641-a435-23d88664daca-config-out\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.585764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.585757 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.585991 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.585780 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.585991 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.585806 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.585991 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.585841 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.585991 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.585872 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.585991 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.585959 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.586209 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.586002 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.586209 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.586047 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-config\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.586209 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.586084 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48998805-0245-4641-a435-23d88664daca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.586209 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.586129 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.586209 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.586200 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.586455 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.586258 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.586455 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.586276 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-web-config\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.586455 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.586294 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.687607 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687564 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-config\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.687607 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687617 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48998805-0245-4641-a435-23d88664daca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.687846 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687668 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.687846 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687720 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.687846 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687755 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.687846 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687779 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-web-config\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.687846 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687799 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.687846 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687830 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688184 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687865 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688184 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687888 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688184 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687911 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/48998805-0245-4641-a435-23d88664daca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688184 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687933 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cfb7\" (UniqueName: \"kubernetes.io/projected/48998805-0245-4641-a435-23d88664daca-kube-api-access-8cfb7\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688184 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687968 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48998805-0245-4641-a435-23d88664daca-config-out\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688184 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.687989 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688184 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.688019 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688184 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.688046 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688184 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.688114 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688184 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.688147 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688184 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.688181 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688688 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.688210 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688688 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.688597 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688791 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.688737 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688845 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.688784 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.688918 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.688895 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.689383 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.689183 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.689856 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.689576 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.692212 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.691253 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-web-config\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.692212 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.691691 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.692212 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.691985 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/48998805-0245-4641-a435-23d88664daca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.692212 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.692022 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.693619 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.693076 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48998805-0245-4641-a435-23d88664daca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.693619 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.693495 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/48998805-0245-4641-a435-23d88664daca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.693619 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.693583 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-config\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.694119 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.694034 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.694519 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.694495 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.695226 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.695202 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48998805-0245-4641-a435-23d88664daca-config-out\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.695509 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.695485 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.695623 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.695486 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.695728 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.695711 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/48998805-0245-4641-a435-23d88664daca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.701417 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.701394 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cfb7\" (UniqueName: \"kubernetes.io/projected/48998805-0245-4641-a435-23d88664daca-kube-api-access-8cfb7\") pod \"prometheus-k8s-0\" (UID: \"48998805-0245-4641-a435-23d88664daca\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.828712 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.828671 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:29.992817 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:29.992782 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 01 00:30:29.996389 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:30:29.996356 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48998805_0245_4641_a435_23d88664daca.slice/crio-9cf1ee2ea2c53ebf27a86616fec3d6615e9206e2cd540425d909bc7fe5dcec7e WatchSource:0}: Error finding container 9cf1ee2ea2c53ebf27a86616fec3d6615e9206e2cd540425d909bc7fe5dcec7e: Status 404 returned error can't find the container with id 9cf1ee2ea2c53ebf27a86616fec3d6615e9206e2cd540425d909bc7fe5dcec7e Aug 01 00:30:30.213670 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:30.213585 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj" event={"ID":"dd7c88f4-a5b2-45bd-9a87-78a902daae7c","Type":"ContainerStarted","Data":"fdddb72c49c2e2c4d83a3ccaf8cf47e416c871799d8979b9dff7f393a9fdd6f9"} Aug 01 00:30:30.214121 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:30.213753 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj" Aug 01 00:30:30.215119 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:30.215088 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48998805-0245-4641-a435-23d88664daca","Type":"ContainerStarted","Data":"9cf1ee2ea2c53ebf27a86616fec3d6615e9206e2cd540425d909bc7fe5dcec7e"} Aug 01 00:30:30.216698 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:30.216674 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" event={"ID":"01a2397b-9138-42f4-b397-4eb2705829d3","Type":"ContainerStarted","Data":"27871be731c6562f28a249ad538adfc26d61b6db9a878fdec911cbbae1f98fdb"} Aug 01 00:30:30.219250 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:30.219231 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj" Aug 01 00:30:30.228637 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:30.228591 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vsshj" podStartSLOduration=1.831327274 podStartE2EDuration="3.228575871s" totalImagesPullingTime="1.397248597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:30.228269504 +0000 UTC m=+119.016991332" watchObservedRunningTime="2026-08-01 00:30:30.228575871 +0000 UTC m=+119.017297697" Aug 01 00:30:30.243783 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:30.243728 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" podStartSLOduration=1.536414509 podStartE2EDuration="3.243712909s" totalImagesPullingTime="1.7072984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:30.242575247 +0000 UTC m=+119.031297067" watchObservedRunningTime="2026-08-01 00:30:30.243712909 +0000 UTC m=+119.032434734" Aug 01 00:30:30.360706 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:30.360674 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fcd8946f6-l8kx8"] Aug 01 00:30:30.364888 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:30.364867 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:30:31.222226 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:31.222137 2563 generic.go:356] "Generic (PLEG): container finished" podID="48998805-0245-4641-a435-23d88664daca" containerID="a829847f90563e0bfe642c6df9fed9a8f3449244e3eb2798b6e5b2b3eb84bd13" exitCode=0 Aug 01 00:30:31.222670 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:31.222216 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48998805-0245-4641-a435-23d88664daca","Type":"ContainerDied","Data":"a829847f90563e0bfe642c6df9fed9a8f3449244e3eb2798b6e5b2b3eb84bd13"} Aug 01 00:30:32.230670 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:32.230621 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" event={"ID":"3f3349b2-3e23-48e3-89d8-76e3fdb6d354","Type":"ContainerStarted","Data":"28fbc27e0b259ed99449387036cca52ad3d64021ddf2be119bbbea96b12a1394"} Aug 01 00:30:32.231211 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:32.230717 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" event={"ID":"3f3349b2-3e23-48e3-89d8-76e3fdb6d354","Type":"ContainerStarted","Data":"f36ac2704640db6304b680e0a3b7f8ef3131575319b1f8199461d113a89ba659"} Aug 01 00:30:32.231211 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:32.230768 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" event={"ID":"3f3349b2-3e23-48e3-89d8-76e3fdb6d354","Type":"ContainerStarted","Data":"47b5ab6c310f629b1b8c1f841cde80c75574b57ecbf821848d5489d3198cd6d5"} Aug 01 00:30:32.252768 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:32.252720 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" podStartSLOduration=1.7665982649999998 podStartE2EDuration="7.252704568s" totalImagesPullingTime="5.486106303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:32.251283877 +0000 UTC m=+121.040005704" watchObservedRunningTime="2026-08-01 00:30:32.252704568 +0000 UTC m=+121.041426391" Aug 01 00:30:33.235364 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:33.235328 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:33.241630 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:33.241605 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-75755945d6-p6b8l" Aug 01 00:30:35.246644 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:35.246550 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48998805-0245-4641-a435-23d88664daca","Type":"ContainerStarted","Data":"dffb6d4c0396c891e750d5939d642a7c48877359bc6a250d4f0708cf511c553e"} Aug 01 00:30:35.246644 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:35.246629 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48998805-0245-4641-a435-23d88664daca","Type":"ContainerStarted","Data":"f6e5008a1da8a2530c47317b7048dddb4d5cf87079907a23f626926b39eca047"} Aug 01 00:30:35.247177 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:35.246657 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48998805-0245-4641-a435-23d88664daca","Type":"ContainerStarted","Data":"57b0ad0acea0f6069adb5e86a15cc1fdab3c5f7835742cf61148f7b8a2391218"} Aug 01 00:30:35.247177 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:35.246689 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48998805-0245-4641-a435-23d88664daca","Type":"ContainerStarted","Data":"a1906b270c2b74869c2686a840f00cdd56414eb68f5b82b96df8ab42d9213746"} Aug 01 00:30:35.247177 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:35.246716 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48998805-0245-4641-a435-23d88664daca","Type":"ContainerStarted","Data":"27f88bce102960231473ba71bde3bc6a0cc038d6714460023202d4ff516312cc"} Aug 01 00:30:35.247177 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:35.246747 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48998805-0245-4641-a435-23d88664daca","Type":"ContainerStarted","Data":"9c74312b848edb8e4a968c35490987742869cea18852384eee3a63b16c99e7e3"} Aug 01 00:30:35.272646 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:35.272603 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.85127339 podStartE2EDuration="6.272588429s" totalImagesPullingTime="4.403208894s" totalInitContainerRuntime="18.106145ms" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:35.271032139 +0000 UTC m=+124.059753979" watchObservedRunningTime="2026-08-01 00:30:35.272588429 +0000 UTC m=+124.061310254" Aug 01 00:30:39.829091 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:39.829033 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:30:41.489962 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:41.489902 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:30:41.492219 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:41.492196 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644d164-f68a-4c37-abd5-cc4f66cbe590-metrics-certs\") pod \"network-metrics-daemon-wkj56\" (UID: \"8644d164-f68a-4c37-abd5-cc4f66cbe590\") " pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:30:41.742561 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:41.742482 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c2x5h\"" Aug 01 00:30:41.750435 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:41.750414 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkj56" Aug 01 00:30:41.878500 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:41.878472 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wkj56"] Aug 01 00:30:41.879872 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:30:41.879850 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8644d164_f68a_4c37_abd5_cc4f66cbe590.slice/crio-9252098fe1cb96a5279e4c078f09cd7a497d49688241845a49a7155d5c3b5f5a WatchSource:0}: Error finding container 9252098fe1cb96a5279e4c078f09cd7a497d49688241845a49a7155d5c3b5f5a: Status 404 returned error can't find the container with id 9252098fe1cb96a5279e4c078f09cd7a497d49688241845a49a7155d5c3b5f5a Aug 01 00:30:42.280292 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:42.280253 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wkj56" event={"ID":"8644d164-f68a-4c37-abd5-cc4f66cbe590","Type":"ContainerStarted","Data":"9252098fe1cb96a5279e4c078f09cd7a497d49688241845a49a7155d5c3b5f5a"} Aug 01 00:30:43.287919 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:43.287823 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wkj56" event={"ID":"8644d164-f68a-4c37-abd5-cc4f66cbe590","Type":"ContainerStarted","Data":"048fa2dc9e15c4a4183707b909016524efec328f3b27117cbf63b756c36ab536"} Aug 01 00:30:43.287919 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:43.287894 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wkj56" event={"ID":"8644d164-f68a-4c37-abd5-cc4f66cbe590","Type":"ContainerStarted","Data":"aa6fcd9abb95f18600963a1da0278a01291701f6ea031e8f0d3b058fbde4cac6"} Aug 01 00:30:43.303636 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:43.303588 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wkj56" podStartSLOduration=130.222695077 podStartE2EDuration="2m11.303573558s" totalImagesPullingTime="1.080878481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:43.302249455 +0000 UTC m=+132.090971320" watchObservedRunningTime="2026-08-01 00:30:43.303573558 +0000 UTC m=+132.092295383" Aug 01 00:30:47.956158 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:47.956111 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:47.956625 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:47.956184 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:30:51.475744 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.475698 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-546bd95747-cvtg4" podUID="c9ad886e-3b36-48fc-806e-2c291e9a553e" containerName="console" containerID="cri-o://e6c7ee73db64667ba14a8c1a9eefe046dbfd08411e83a804b455b6063401ac2d" gracePeriod=15 Aug 01 00:30:51.713846 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.713822 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:51.778557 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.778488 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-config\") pod \"c9ad886e-3b36-48fc-806e-2c291e9a553e\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " Aug 01 00:30:51.778557 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.778539 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-oauth-config\") pod \"c9ad886e-3b36-48fc-806e-2c291e9a553e\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " Aug 01 00:30:51.778761 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.778575 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-oauth-serving-cert\") pod \"c9ad886e-3b36-48fc-806e-2c291e9a553e\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " Aug 01 00:30:51.778761 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.778591 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb96t\" (UniqueName: \"kubernetes.io/projected/c9ad886e-3b36-48fc-806e-2c291e9a553e-kube-api-access-zb96t\") pod \"c9ad886e-3b36-48fc-806e-2c291e9a553e\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " Aug 01 00:30:51.778761 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.778622 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-service-ca\") pod \"c9ad886e-3b36-48fc-806e-2c291e9a553e\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " Aug 01 00:30:51.778916 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.778797 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-serving-cert\") pod \"c9ad886e-3b36-48fc-806e-2c291e9a553e\" (UID: \"c9ad886e-3b36-48fc-806e-2c291e9a553e\") " Aug 01 00:30:51.778984 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.778961 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-config" (OuterVolumeSpecName: "console-config") pod "c9ad886e-3b36-48fc-806e-2c291e9a553e" (UID: "c9ad886e-3b36-48fc-806e-2c291e9a553e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:30:51.779077 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.779021 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c9ad886e-3b36-48fc-806e-2c291e9a553e" (UID: "c9ad886e-3b36-48fc-806e-2c291e9a553e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:30:51.779161 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.779032 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-service-ca" (OuterVolumeSpecName: "service-ca") pod "c9ad886e-3b36-48fc-806e-2c291e9a553e" (UID: "c9ad886e-3b36-48fc-806e-2c291e9a553e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:30:51.779229 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.779211 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-config\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:51.779274 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.779236 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-oauth-serving-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:51.780828 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.780795 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c9ad886e-3b36-48fc-806e-2c291e9a553e" (UID: "c9ad886e-3b36-48fc-806e-2c291e9a553e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:30:51.780923 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.780834 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ad886e-3b36-48fc-806e-2c291e9a553e-kube-api-access-zb96t" (OuterVolumeSpecName: "kube-api-access-zb96t") pod "c9ad886e-3b36-48fc-806e-2c291e9a553e" (UID: "c9ad886e-3b36-48fc-806e-2c291e9a553e"). InnerVolumeSpecName "kube-api-access-zb96t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:30:51.780923 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.780832 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c9ad886e-3b36-48fc-806e-2c291e9a553e" (UID: "c9ad886e-3b36-48fc-806e-2c291e9a553e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:30:51.880510 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.880478 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-oauth-config\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:51.880510 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.880506 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb96t\" (UniqueName: \"kubernetes.io/projected/c9ad886e-3b36-48fc-806e-2c291e9a553e-kube-api-access-zb96t\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:51.880510 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.880516 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ad886e-3b36-48fc-806e-2c291e9a553e-service-ca\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:51.880729 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:51.880525 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9ad886e-3b36-48fc-806e-2c291e9a553e-console-serving-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:52.325672 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:52.325636 2563 generic.go:356] "Generic (PLEG): container finished" podID="c9ad886e-3b36-48fc-806e-2c291e9a553e" containerID="e6c7ee73db64667ba14a8c1a9eefe046dbfd08411e83a804b455b6063401ac2d" exitCode=0 Aug 01 00:30:52.325822 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:52.325683 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546bd95747-cvtg4" Aug 01 00:30:52.325822 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:52.325691 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546bd95747-cvtg4" event={"ID":"c9ad886e-3b36-48fc-806e-2c291e9a553e","Type":"ContainerDied","Data":"e6c7ee73db64667ba14a8c1a9eefe046dbfd08411e83a804b455b6063401ac2d"} Aug 01 00:30:52.325822 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:52.325788 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546bd95747-cvtg4" event={"ID":"c9ad886e-3b36-48fc-806e-2c291e9a553e","Type":"ContainerDied","Data":"1a578a6ef956bb4d3e81a3c1bf0579f85196e0d76471e85ea70203d3781eae9c"} Aug 01 00:30:52.325822 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:52.325817 2563 scope.go:117] "RemoveContainer" containerID="e6c7ee73db64667ba14a8c1a9eefe046dbfd08411e83a804b455b6063401ac2d" Aug 01 00:30:52.334246 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:52.334228 2563 scope.go:117] "RemoveContainer" containerID="e6c7ee73db64667ba14a8c1a9eefe046dbfd08411e83a804b455b6063401ac2d" Aug 01 00:30:52.334497 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:30:52.334477 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c7ee73db64667ba14a8c1a9eefe046dbfd08411e83a804b455b6063401ac2d\": container with ID starting with e6c7ee73db64667ba14a8c1a9eefe046dbfd08411e83a804b455b6063401ac2d not found: ID does not exist" containerID="e6c7ee73db64667ba14a8c1a9eefe046dbfd08411e83a804b455b6063401ac2d" Aug 01 00:30:52.334544 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:52.334505 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c7ee73db64667ba14a8c1a9eefe046dbfd08411e83a804b455b6063401ac2d"} err="failed to get container status \"e6c7ee73db64667ba14a8c1a9eefe046dbfd08411e83a804b455b6063401ac2d\": rpc error: code = NotFound desc = could not find container \"e6c7ee73db64667ba14a8c1a9eefe046dbfd08411e83a804b455b6063401ac2d\": container with ID starting with e6c7ee73db64667ba14a8c1a9eefe046dbfd08411e83a804b455b6063401ac2d not found: ID does not exist" Aug 01 00:30:52.346085 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:52.346035 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546bd95747-cvtg4"] Aug 01 00:30:52.347848 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:52.347827 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-546bd95747-cvtg4"] Aug 01 00:30:53.726897 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:53.726861 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9ad886e-3b36-48fc-806e-2c291e9a553e" path="/var/lib/kubelet/pods/c9ad886e-3b36-48fc-806e-2c291e9a553e/volumes" Aug 01 00:30:55.380191 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.380152 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" podUID="e5dc1c37-a456-45c8-a01c-c30082421e3b" containerName="registry" containerID="cri-o://f3e6faf5381eecec99dcb2352ea7af1598cd72a19c146c2865228237db5bcc58" gracePeriod=30 Aug 01 00:30:55.638355 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.638303 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:30:55.712668 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.712638 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5dc1c37-a456-45c8-a01c-c30082421e3b-installation-pull-secrets\") pod \"e5dc1c37-a456-45c8-a01c-c30082421e3b\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " Aug 01 00:30:55.712668 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.712672 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c5ns\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-kube-api-access-4c5ns\") pod \"e5dc1c37-a456-45c8-a01c-c30082421e3b\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " Aug 01 00:30:55.712862 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.712699 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-bound-sa-token\") pod \"e5dc1c37-a456-45c8-a01c-c30082421e3b\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " Aug 01 00:30:55.712862 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.712736 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5dc1c37-a456-45c8-a01c-c30082421e3b-image-registry-private-configuration\") pod \"e5dc1c37-a456-45c8-a01c-c30082421e3b\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " Aug 01 00:30:55.712862 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.712846 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5dc1c37-a456-45c8-a01c-c30082421e3b-ca-trust-extracted\") pod \"e5dc1c37-a456-45c8-a01c-c30082421e3b\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " Aug 01 00:30:55.712989 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.712885 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5dc1c37-a456-45c8-a01c-c30082421e3b-trusted-ca\") pod \"e5dc1c37-a456-45c8-a01c-c30082421e3b\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " Aug 01 00:30:55.712989 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.712911 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-certificates\") pod \"e5dc1c37-a456-45c8-a01c-c30082421e3b\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " Aug 01 00:30:55.712989 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.712979 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls\") pod \"e5dc1c37-a456-45c8-a01c-c30082421e3b\" (UID: \"e5dc1c37-a456-45c8-a01c-c30082421e3b\") " Aug 01 00:30:55.713412 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.713362 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5dc1c37-a456-45c8-a01c-c30082421e3b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e5dc1c37-a456-45c8-a01c-c30082421e3b" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:30:55.713514 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.713478 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e5dc1c37-a456-45c8-a01c-c30082421e3b" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:30:55.715208 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.715173 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5dc1c37-a456-45c8-a01c-c30082421e3b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e5dc1c37-a456-45c8-a01c-c30082421e3b" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:30:55.715477 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.715439 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5dc1c37-a456-45c8-a01c-c30082421e3b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e5dc1c37-a456-45c8-a01c-c30082421e3b" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:30:55.715477 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.715454 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e5dc1c37-a456-45c8-a01c-c30082421e3b" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:30:55.715635 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.715490 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-kube-api-access-4c5ns" (OuterVolumeSpecName: "kube-api-access-4c5ns") pod "e5dc1c37-a456-45c8-a01c-c30082421e3b" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b"). InnerVolumeSpecName "kube-api-access-4c5ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:30:55.715635 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.715571 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e5dc1c37-a456-45c8-a01c-c30082421e3b" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:30:55.721484 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.721463 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5dc1c37-a456-45c8-a01c-c30082421e3b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e5dc1c37-a456-45c8-a01c-c30082421e3b" (UID: "e5dc1c37-a456-45c8-a01c-c30082421e3b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:30:55.814600 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.814567 2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5dc1c37-a456-45c8-a01c-c30082421e3b-installation-pull-secrets\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:55.814600 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.814593 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c5ns\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-kube-api-access-4c5ns\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:55.814600 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.814603 2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-bound-sa-token\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:55.814780 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.814613 2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5dc1c37-a456-45c8-a01c-c30082421e3b-image-registry-private-configuration\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:55.814780 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.814622 2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5dc1c37-a456-45c8-a01c-c30082421e3b-ca-trust-extracted\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:55.814780 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.814631 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5dc1c37-a456-45c8-a01c-c30082421e3b-trusted-ca\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:55.814780 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.814639 2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-certificates\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:55.814780 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:55.814647 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5dc1c37-a456-45c8-a01c-c30082421e3b-registry-tls\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:30:56.344237 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:56.344200 2563 generic.go:356] "Generic (PLEG): container finished" podID="e5dc1c37-a456-45c8-a01c-c30082421e3b" containerID="f3e6faf5381eecec99dcb2352ea7af1598cd72a19c146c2865228237db5bcc58" exitCode=0 Aug 01 00:30:56.344426 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:56.344248 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" Aug 01 00:30:56.344426 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:56.344282 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" event={"ID":"e5dc1c37-a456-45c8-a01c-c30082421e3b","Type":"ContainerDied","Data":"f3e6faf5381eecec99dcb2352ea7af1598cd72a19c146c2865228237db5bcc58"} Aug 01 00:30:56.344426 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:56.344334 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcd8946f6-l8kx8" event={"ID":"e5dc1c37-a456-45c8-a01c-c30082421e3b","Type":"ContainerDied","Data":"619c87143c32c9526fafa7c4e485d51bb3b88f65eb40d80c0334fad22e09c0d2"} Aug 01 00:30:56.344426 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:56.344353 2563 scope.go:117] "RemoveContainer" containerID="f3e6faf5381eecec99dcb2352ea7af1598cd72a19c146c2865228237db5bcc58" Aug 01 00:30:56.352179 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:56.352161 2563 scope.go:117] "RemoveContainer" containerID="f3e6faf5381eecec99dcb2352ea7af1598cd72a19c146c2865228237db5bcc58" Aug 01 00:30:56.352414 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:30:56.352396 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e6faf5381eecec99dcb2352ea7af1598cd72a19c146c2865228237db5bcc58\": container with ID starting with f3e6faf5381eecec99dcb2352ea7af1598cd72a19c146c2865228237db5bcc58 not found: ID does not exist" containerID="f3e6faf5381eecec99dcb2352ea7af1598cd72a19c146c2865228237db5bcc58" Aug 01 00:30:56.352482 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:56.352423 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e6faf5381eecec99dcb2352ea7af1598cd72a19c146c2865228237db5bcc58"} err="failed to get container status \"f3e6faf5381eecec99dcb2352ea7af1598cd72a19c146c2865228237db5bcc58\": rpc error: code = NotFound desc = could not find container \"f3e6faf5381eecec99dcb2352ea7af1598cd72a19c146c2865228237db5bcc58\": container with ID starting with f3e6faf5381eecec99dcb2352ea7af1598cd72a19c146c2865228237db5bcc58 not found: ID does not exist" Aug 01 00:30:56.387433 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:56.387402 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fcd8946f6-l8kx8"] Aug 01 00:30:56.399697 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:56.399672 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6fcd8946f6-l8kx8"] Aug 01 00:30:57.726816 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:30:57.726780 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5dc1c37-a456-45c8-a01c-c30082421e3b" path="/var/lib/kubelet/pods/e5dc1c37-a456-45c8-a01c-c30082421e3b/volumes" Aug 01 00:31:05.381354 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:05.381322 2563 generic.go:356] "Generic (PLEG): container finished" podID="a878cee9-835b-4d43-9d83-b5469f709610" containerID="7d306e359dede0d9dfaa03cb1b0c7e413f7b753f24396771f30cc0da3d5e47f2" exitCode=0 Aug 01 00:31:05.381703 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:05.381379 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-nlksz" event={"ID":"a878cee9-835b-4d43-9d83-b5469f709610","Type":"ContainerDied","Data":"7d306e359dede0d9dfaa03cb1b0c7e413f7b753f24396771f30cc0da3d5e47f2"} Aug 01 00:31:05.381703 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:05.381683 2563 scope.go:117] "RemoveContainer" containerID="7d306e359dede0d9dfaa03cb1b0c7e413f7b753f24396771f30cc0da3d5e47f2" Aug 01 00:31:06.387268 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:06.387233 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-nlksz" event={"ID":"a878cee9-835b-4d43-9d83-b5469f709610","Type":"ContainerStarted","Data":"4792aa33e106f8c952dc491d385d8e2e4bb221ecde24ccd67cfd1f519789d66b"} Aug 01 00:31:07.961644 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:07.961610 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:31:07.965481 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:07.965457 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-65d6c48b96-kr2xn" Aug 01 00:31:10.406264 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:10.406234 2563 generic.go:356] "Generic (PLEG): container finished" podID="bcacc184-28ea-4d69-a87c-b7c08c2add99" containerID="170e6104a95d8bfff9880e5fdcca87d3cab9b25da9cdb8f64985755eb1bf385c" exitCode=0 Aug 01 00:31:10.406668 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:10.406319 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" event={"ID":"bcacc184-28ea-4d69-a87c-b7c08c2add99","Type":"ContainerDied","Data":"170e6104a95d8bfff9880e5fdcca87d3cab9b25da9cdb8f64985755eb1bf385c"} Aug 01 00:31:10.406668 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:10.406602 2563 scope.go:117] "RemoveContainer" containerID="170e6104a95d8bfff9880e5fdcca87d3cab9b25da9cdb8f64985755eb1bf385c" Aug 01 00:31:11.411400 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:11.411368 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6fc4k" event={"ID":"bcacc184-28ea-4d69-a87c-b7c08c2add99","Type":"ContainerStarted","Data":"f5558c3412e3f89e755aedf555a875c0ba775a75f3bdc215141ad2b3b0cedc8a"} Aug 01 00:31:29.829186 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:29.829114 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:31:29.847691 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:29.847666 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:31:30.513712 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:30.513676 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 01 00:31:43.546682 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:43.546660 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:31:43.547037 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:43.546698 2563 generic.go:356] "Generic (PLEG): container finished" podID="802cbf7b-10e0-4ff6-8443-c0f7b60a9727" containerID="619272bd3741164b38da55f0485150de5de6dedea5629fcc5a7f13eb982fd006" exitCode=2 Aug 01 00:31:43.547037 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:43.546723 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" event={"ID":"802cbf7b-10e0-4ff6-8443-c0f7b60a9727","Type":"ContainerDied","Data":"619272bd3741164b38da55f0485150de5de6dedea5629fcc5a7f13eb982fd006"} Aug 01 00:31:43.547037 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:43.547001 2563 scope.go:117] "RemoveContainer" containerID="619272bd3741164b38da55f0485150de5de6dedea5629fcc5a7f13eb982fd006" Aug 01 00:31:44.551933 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:44.551905 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:31:44.552322 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:31:44.551981 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-znmsh" event={"ID":"802cbf7b-10e0-4ff6-8443-c0f7b60a9727","Type":"ContainerStarted","Data":"cece3bdce631c71d4991c68673e6802354fd5723b24b5db4b74dfee0ce27cfee"} Aug 01 00:33:12.565768 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.565737 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dvqdd"] Aug 01 00:33:12.566190 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.566070 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9ad886e-3b36-48fc-806e-2c291e9a553e" containerName="console" Aug 01 00:33:12.566190 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.566082 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9ad886e-3b36-48fc-806e-2c291e9a553e" containerName="console" Aug 01 00:33:12.566190 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.566091 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5dc1c37-a456-45c8-a01c-c30082421e3b" containerName="registry" Aug 01 00:33:12.566190 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.566096 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5dc1c37-a456-45c8-a01c-c30082421e3b" containerName="registry" Aug 01 00:33:12.566190 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.566173 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5dc1c37-a456-45c8-a01c-c30082421e3b" containerName="registry" Aug 01 00:33:12.566190 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.566181 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9ad886e-3b36-48fc-806e-2c291e9a553e" containerName="console" Aug 01 00:33:12.569097 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.569081 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dvqdd" Aug 01 00:33:12.571449 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.571427 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 01 00:33:12.571557 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.571433 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ngbcw\"" Aug 01 00:33:12.572605 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.572591 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 01 00:33:12.577917 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.577897 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dvqdd"] Aug 01 00:33:12.653161 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.653129 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl4vj\" (UniqueName: \"kubernetes.io/projected/e7a68699-a580-43be-be1b-d7da824d6de4-kube-api-access-nl4vj\") pod \"cert-manager-55b4d954d5-dvqdd\" (UID: \"e7a68699-a580-43be-be1b-d7da824d6de4\") " pod="cert-manager/cert-manager-55b4d954d5-dvqdd" Aug 01 00:33:12.653264 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.653181 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7a68699-a580-43be-be1b-d7da824d6de4-bound-sa-token\") pod \"cert-manager-55b4d954d5-dvqdd\" (UID: \"e7a68699-a580-43be-be1b-d7da824d6de4\") " pod="cert-manager/cert-manager-55b4d954d5-dvqdd" Aug 01 00:33:12.754258 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.754223 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7a68699-a580-43be-be1b-d7da824d6de4-bound-sa-token\") pod \"cert-manager-55b4d954d5-dvqdd\" (UID: \"e7a68699-a580-43be-be1b-d7da824d6de4\") " pod="cert-manager/cert-manager-55b4d954d5-dvqdd" Aug 01 00:33:12.754367 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.754323 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl4vj\" (UniqueName: \"kubernetes.io/projected/e7a68699-a580-43be-be1b-d7da824d6de4-kube-api-access-nl4vj\") pod \"cert-manager-55b4d954d5-dvqdd\" (UID: \"e7a68699-a580-43be-be1b-d7da824d6de4\") " pod="cert-manager/cert-manager-55b4d954d5-dvqdd" Aug 01 00:33:12.761800 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.761774 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7a68699-a580-43be-be1b-d7da824d6de4-bound-sa-token\") pod \"cert-manager-55b4d954d5-dvqdd\" (UID: \"e7a68699-a580-43be-be1b-d7da824d6de4\") " pod="cert-manager/cert-manager-55b4d954d5-dvqdd" Aug 01 00:33:12.762097 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.762075 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl4vj\" (UniqueName: \"kubernetes.io/projected/e7a68699-a580-43be-be1b-d7da824d6de4-kube-api-access-nl4vj\") pod \"cert-manager-55b4d954d5-dvqdd\" (UID: \"e7a68699-a580-43be-be1b-d7da824d6de4\") " pod="cert-manager/cert-manager-55b4d954d5-dvqdd" Aug 01 00:33:12.879413 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:12.879346 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dvqdd" Aug 01 00:33:13.000896 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:13.000863 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dvqdd"] Aug 01 00:33:13.003742 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:33:13.003717 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7a68699_a580_43be_be1b_d7da824d6de4.slice/crio-dd4db9d62d5a7c9b37b88cd21e2611c5dc90ceaf826a5ce8e3967a5159c14c4c WatchSource:0}: Error finding container dd4db9d62d5a7c9b37b88cd21e2611c5dc90ceaf826a5ce8e3967a5159c14c4c: Status 404 returned error can't find the container with id dd4db9d62d5a7c9b37b88cd21e2611c5dc90ceaf826a5ce8e3967a5159c14c4c Aug 01 00:33:13.890618 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:13.890555 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dvqdd" event={"ID":"e7a68699-a580-43be-be1b-d7da824d6de4","Type":"ContainerStarted","Data":"dd4db9d62d5a7c9b37b88cd21e2611c5dc90ceaf826a5ce8e3967a5159c14c4c"} Aug 01 00:33:16.908071 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:16.908025 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dvqdd" event={"ID":"e7a68699-a580-43be-be1b-d7da824d6de4","Type":"ContainerStarted","Data":"76e16fdcf39889c505b76f32523f92adc7ffffe4571fd40c7c160a7088d0a1f0"} Aug 01 00:33:16.929719 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:16.929673 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dvqdd" podStartSLOduration=1.747372562 podStartE2EDuration="4.929658386s" totalImagesPullingTime="3.182285824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:33:16.928090231 +0000 UTC m=+285.716812058" watchObservedRunningTime="2026-08-01 00:33:16.929658386 +0000 UTC m=+285.718380213" Aug 01 00:33:31.628266 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:31.628237 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:33:31.632618 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:31.632587 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:33:31.634718 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:31.634698 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:33:31.639526 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:31.639503 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:33:31.641354 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:31.641334 2563 kubelet.go:1648] "Image garbage collection succeeded" Aug 01 00:33:35.881583 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:35.881550 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp"] Aug 01 00:33:35.888433 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:35.888412 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:35.891246 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:35.891093 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 01 00:33:35.891246 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:35.891109 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 01 00:33:35.891246 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:35.891179 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8n8f8\"" Aug 01 00:33:35.891507 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:35.891290 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 01 00:33:35.891507 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:35.891387 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 01 00:33:35.898780 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:35.898758 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp"] Aug 01 00:33:36.031998 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.031969 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-manifests\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.031998 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.032001 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.032216 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.032029 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgvbz\" (UniqueName: \"kubernetes.io/projected/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-kube-api-access-kgvbz\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.032216 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.032114 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-tmp\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.032216 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.032176 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.133097 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.133000 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-manifests\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.133097 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.133032 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.133097 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.133074 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgvbz\" (UniqueName: \"kubernetes.io/projected/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-kube-api-access-kgvbz\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.133349 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.133111 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-tmp\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.133349 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.133174 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.133465 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.133394 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-manifests\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.135304 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.135271 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-tmp\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.135697 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.135676 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.135743 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.135676 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.142192 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.142168 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgvbz\" (UniqueName: \"kubernetes.io/projected/27a0d7fa-1f37-4099-8e34-cede1ae3dba2-kube-api-access-kgvbz\") pod \"opendatahub-operator-controller-manager-6c56689d88-dgxwp\" (UID: \"27a0d7fa-1f37-4099-8e34-cede1ae3dba2\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.199957 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.199935 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:36.336671 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.336641 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp"] Aug 01 00:33:36.339975 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:33:36.339948 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a0d7fa_1f37_4099_8e34_cede1ae3dba2.slice/crio-b4589247361de1ffe9fa13abf2e630da83ccc3764ad31db1d7af45e61f096b47 WatchSource:0}: Error finding container b4589247361de1ffe9fa13abf2e630da83ccc3764ad31db1d7af45e61f096b47: Status 404 returned error can't find the container with id b4589247361de1ffe9fa13abf2e630da83ccc3764ad31db1d7af45e61f096b47 Aug 01 00:33:36.341851 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.341833 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 00:33:36.990044 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:36.990003 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" event={"ID":"27a0d7fa-1f37-4099-8e34-cede1ae3dba2","Type":"ContainerStarted","Data":"b4589247361de1ffe9fa13abf2e630da83ccc3764ad31db1d7af45e61f096b47"} Aug 01 00:33:40.006229 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:40.006154 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" event={"ID":"27a0d7fa-1f37-4099-8e34-cede1ae3dba2","Type":"ContainerStarted","Data":"f745045abdb7b3c81a5b8d636e573e05904d369631f28aacd4f916e72ea9a35a"} Aug 01 00:33:41.011986 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:41.011955 2563 generic.go:356] "Generic (PLEG): container finished" podID="27a0d7fa-1f37-4099-8e34-cede1ae3dba2" containerID="f745045abdb7b3c81a5b8d636e573e05904d369631f28aacd4f916e72ea9a35a" exitCode=0 Aug 01 00:33:41.012486 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:41.012018 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" event={"ID":"27a0d7fa-1f37-4099-8e34-cede1ae3dba2","Type":"ContainerDied","Data":"f745045abdb7b3c81a5b8d636e573e05904d369631f28aacd4f916e72ea9a35a"} Aug 01 00:33:42.018990 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:42.018952 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" event={"ID":"27a0d7fa-1f37-4099-8e34-cede1ae3dba2","Type":"ContainerStarted","Data":"efa2b7578ce077d1524787978d40d87cf95719836b64be9634187d9e4215e022"} Aug 01 00:33:42.019466 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:42.019048 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:42.043074 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:42.043013 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" podStartSLOduration=2.271824813 podStartE2EDuration="7.042994634s" totalImagesPullingTime="3.715807728s" totalInitContainerRuntime="1.055362093s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:33:42.041120412 +0000 UTC m=+310.829842238" watchObservedRunningTime="2026-08-01 00:33:42.042994634 +0000 UTC m=+310.831716462" Aug 01 00:33:53.025924 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:53.025894 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-dgxwp" Aug 01 00:33:55.526693 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.526665 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq"] Aug 01 00:33:55.529987 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.529968 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.535247 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.535221 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 01 00:33:55.536749 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.536244 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 01 00:33:55.536749 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.536342 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:33:55.536749 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.536396 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 01 00:33:55.536749 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.536441 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-t6hqt\"" Aug 01 00:33:55.536749 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.536535 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 01 00:33:55.541854 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.541834 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq"] Aug 01 00:33:55.688213 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.688180 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/074fa142-2156-4b3b-937b-c9ed2b811e4b-cert\") pod \"lws-controller-manager-5dd57cb99d-tpqcq\" (UID: \"074fa142-2156-4b3b-937b-c9ed2b811e4b\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.688392 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.688243 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/074fa142-2156-4b3b-937b-c9ed2b811e4b-manager-config\") pod \"lws-controller-manager-5dd57cb99d-tpqcq\" (UID: \"074fa142-2156-4b3b-937b-c9ed2b811e4b\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.688392 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.688313 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/074fa142-2156-4b3b-937b-c9ed2b811e4b-metrics-cert\") pod \"lws-controller-manager-5dd57cb99d-tpqcq\" (UID: \"074fa142-2156-4b3b-937b-c9ed2b811e4b\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.688392 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.688352 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zslpz\" (UniqueName: \"kubernetes.io/projected/074fa142-2156-4b3b-937b-c9ed2b811e4b-kube-api-access-zslpz\") pod \"lws-controller-manager-5dd57cb99d-tpqcq\" (UID: \"074fa142-2156-4b3b-937b-c9ed2b811e4b\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.789716 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.789637 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/074fa142-2156-4b3b-937b-c9ed2b811e4b-metrics-cert\") pod \"lws-controller-manager-5dd57cb99d-tpqcq\" (UID: \"074fa142-2156-4b3b-937b-c9ed2b811e4b\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.789716 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.789677 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zslpz\" (UniqueName: \"kubernetes.io/projected/074fa142-2156-4b3b-937b-c9ed2b811e4b-kube-api-access-zslpz\") pod \"lws-controller-manager-5dd57cb99d-tpqcq\" (UID: \"074fa142-2156-4b3b-937b-c9ed2b811e4b\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.789716 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.789715 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/074fa142-2156-4b3b-937b-c9ed2b811e4b-cert\") pod \"lws-controller-manager-5dd57cb99d-tpqcq\" (UID: \"074fa142-2156-4b3b-937b-c9ed2b811e4b\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.789946 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.789742 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/074fa142-2156-4b3b-937b-c9ed2b811e4b-manager-config\") pod \"lws-controller-manager-5dd57cb99d-tpqcq\" (UID: \"074fa142-2156-4b3b-937b-c9ed2b811e4b\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.790427 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.790406 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/074fa142-2156-4b3b-937b-c9ed2b811e4b-manager-config\") pod \"lws-controller-manager-5dd57cb99d-tpqcq\" (UID: \"074fa142-2156-4b3b-937b-c9ed2b811e4b\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.792091 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.792049 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/074fa142-2156-4b3b-937b-c9ed2b811e4b-metrics-cert\") pod \"lws-controller-manager-5dd57cb99d-tpqcq\" (UID: \"074fa142-2156-4b3b-937b-c9ed2b811e4b\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.792211 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.792191 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/074fa142-2156-4b3b-937b-c9ed2b811e4b-cert\") pod \"lws-controller-manager-5dd57cb99d-tpqcq\" (UID: \"074fa142-2156-4b3b-937b-c9ed2b811e4b\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.802905 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.802884 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zslpz\" (UniqueName: \"kubernetes.io/projected/074fa142-2156-4b3b-937b-c9ed2b811e4b-kube-api-access-zslpz\") pod \"lws-controller-manager-5dd57cb99d-tpqcq\" (UID: \"074fa142-2156-4b3b-937b-c9ed2b811e4b\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.838863 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.838835 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:55.976933 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:55.976909 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq"] Aug 01 00:33:55.978999 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:33:55.978970 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074fa142_2156_4b3b_937b_c9ed2b811e4b.slice/crio-a901bd2422cd59abec9a0a66bf004aac7343e2551d3da7edce641ebead9aee0a WatchSource:0}: Error finding container a901bd2422cd59abec9a0a66bf004aac7343e2551d3da7edce641ebead9aee0a: Status 404 returned error can't find the container with id a901bd2422cd59abec9a0a66bf004aac7343e2551d3da7edce641ebead9aee0a Aug 01 00:33:56.078419 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:56.078386 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" event={"ID":"074fa142-2156-4b3b-937b-c9ed2b811e4b","Type":"ContainerStarted","Data":"a901bd2422cd59abec9a0a66bf004aac7343e2551d3da7edce641ebead9aee0a"} Aug 01 00:33:59.095316 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:59.095272 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" event={"ID":"074fa142-2156-4b3b-937b-c9ed2b811e4b","Type":"ContainerStarted","Data":"ab06233bdcaa26395ede042af90d7caed0801939d196da2132bfc2bac76b8869"} Aug 01 00:33:59.095679 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:59.095419 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:33:59.111618 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:33:59.111568 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" podStartSLOduration=1.174419213 podStartE2EDuration="4.111552369s" totalImagesPullingTime="2.937133156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:33:59.110401018 +0000 UTC m=+327.899122855" watchObservedRunningTime="2026-08-01 00:33:59.111552369 +0000 UTC m=+327.900274283" Aug 01 00:34:02.960697 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:02.960663 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv"] Aug 01 00:34:02.964856 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:02.964837 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:02.968043 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:02.967714 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 01 00:34:02.968043 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:02.967779 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 01 00:34:02.968043 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:02.967793 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 01 00:34:02.968043 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:02.967937 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 01 00:34:02.968309 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:02.968074 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-g2w66\"" Aug 01 00:34:02.968309 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:02.968146 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 01 00:34:02.968309 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:02.968154 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 01 00:34:02.981749 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:02.981726 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv"] Aug 01 00:34:03.160080 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.160028 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c28f4c7-2878-4ef2-819c-c72cf78034d1-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.160080 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.160085 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c28f4c7-2878-4ef2-819c-c72cf78034d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.160269 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.160159 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c28f4c7-2878-4ef2-819c-c72cf78034d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.160269 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.160238 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c28f4c7-2878-4ef2-819c-c72cf78034d1-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.160269 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.160262 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c28f4c7-2878-4ef2-819c-c72cf78034d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.160364 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.160282 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktpn\" (UniqueName: \"kubernetes.io/projected/3c28f4c7-2878-4ef2-819c-c72cf78034d1-kube-api-access-sktpn\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.160364 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.160298 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c28f4c7-2878-4ef2-819c-c72cf78034d1-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.261354 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.261274 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c28f4c7-2878-4ef2-819c-c72cf78034d1-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.261354 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.261311 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c28f4c7-2878-4ef2-819c-c72cf78034d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.261354 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.261330 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktpn\" (UniqueName: \"kubernetes.io/projected/3c28f4c7-2878-4ef2-819c-c72cf78034d1-kube-api-access-sktpn\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.261354 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.261346 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c28f4c7-2878-4ef2-819c-c72cf78034d1-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.261673 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.261467 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c28f4c7-2878-4ef2-819c-c72cf78034d1-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.261673 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.261507 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c28f4c7-2878-4ef2-819c-c72cf78034d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.261673 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.261553 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c28f4c7-2878-4ef2-819c-c72cf78034d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.262257 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.262220 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c28f4c7-2878-4ef2-819c-c72cf78034d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.263850 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.263824 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c28f4c7-2878-4ef2-819c-c72cf78034d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.263952 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.263885 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c28f4c7-2878-4ef2-819c-c72cf78034d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.264132 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.264115 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c28f4c7-2878-4ef2-819c-c72cf78034d1-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.264271 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.264251 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c28f4c7-2878-4ef2-819c-c72cf78034d1-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.271547 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.271501 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c28f4c7-2878-4ef2-819c-c72cf78034d1-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.272148 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.272127 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktpn\" (UniqueName: \"kubernetes.io/projected/3c28f4c7-2878-4ef2-819c-c72cf78034d1-kube-api-access-sktpn\") pod \"istiod-openshift-gateway-5677ccb49b-rc6mv\" (UID: \"3c28f4c7-2878-4ef2-819c-c72cf78034d1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.274877 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.274853 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:03.413707 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:03.413675 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv"] Aug 01 00:34:03.416049 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:34:03.416022 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c28f4c7_2878_4ef2_819c_c72cf78034d1.slice/crio-faa3dcfe6bda0206a306666efb10814739418014925db5c2d24f313e15c3ee89 WatchSource:0}: Error finding container faa3dcfe6bda0206a306666efb10814739418014925db5c2d24f313e15c3ee89: Status 404 returned error can't find the container with id faa3dcfe6bda0206a306666efb10814739418014925db5c2d24f313e15c3ee89 Aug 01 00:34:04.120553 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:04.120509 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" event={"ID":"3c28f4c7-2878-4ef2-819c-c72cf78034d1","Type":"ContainerStarted","Data":"faa3dcfe6bda0206a306666efb10814739418014925db5c2d24f313e15c3ee89"} Aug 01 00:34:05.956586 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:05.956541 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:34:05.957071 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:05.956630 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:34:06.132352 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:06.132311 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" event={"ID":"3c28f4c7-2878-4ef2-819c-c72cf78034d1","Type":"ContainerStarted","Data":"46828da19e449c2df0a77f8a45dfe566d6c95226d921331fa1cbeb6d05768554"} Aug 01 00:34:06.132778 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:06.132459 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:06.157363 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:06.157316 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" podStartSLOduration=1.619086128 podStartE2EDuration="4.157298997s" totalImagesPullingTime="2.538212869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:06.155195071 +0000 UTC m=+334.943916919" watchObservedRunningTime="2026-08-01 00:34:06.157298997 +0000 UTC m=+334.946020823" Aug 01 00:34:07.138284 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:07.138256 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rc6mv" Aug 01 00:34:10.101433 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:34:10.101402 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-tpqcq" Aug 01 00:35:46.821734 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:46.821700 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg"] Aug 01 00:35:46.825019 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:46.825001 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" Aug 01 00:35:46.827555 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:46.827534 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 01 00:35:46.827555 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:46.827549 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 01 00:35:46.827692 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:46.827534 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dnhvv\"" Aug 01 00:35:46.837945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:46.837923 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg"] Aug 01 00:35:46.931127 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:46.931101 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kcsp\" (UniqueName: \"kubernetes.io/projected/a582a43b-f0c6-40d4-8550-80a7f1dba309-kube-api-access-2kcsp\") pod \"kuadrant-operator-controller-manager-68cf589589-852cg\" (UID: \"a582a43b-f0c6-40d4-8550-80a7f1dba309\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" Aug 01 00:35:46.931248 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:46.931144 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a582a43b-f0c6-40d4-8550-80a7f1dba309-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-852cg\" (UID: \"a582a43b-f0c6-40d4-8550-80a7f1dba309\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" Aug 01 00:35:47.032292 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:47.032257 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a582a43b-f0c6-40d4-8550-80a7f1dba309-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-852cg\" (UID: \"a582a43b-f0c6-40d4-8550-80a7f1dba309\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" Aug 01 00:35:47.032459 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:47.032372 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kcsp\" (UniqueName: \"kubernetes.io/projected/a582a43b-f0c6-40d4-8550-80a7f1dba309-kube-api-access-2kcsp\") pod \"kuadrant-operator-controller-manager-68cf589589-852cg\" (UID: \"a582a43b-f0c6-40d4-8550-80a7f1dba309\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" Aug 01 00:35:47.032638 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:47.032618 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a582a43b-f0c6-40d4-8550-80a7f1dba309-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-852cg\" (UID: \"a582a43b-f0c6-40d4-8550-80a7f1dba309\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" Aug 01 00:35:47.040381 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:47.040357 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kcsp\" (UniqueName: \"kubernetes.io/projected/a582a43b-f0c6-40d4-8550-80a7f1dba309-kube-api-access-2kcsp\") pod \"kuadrant-operator-controller-manager-68cf589589-852cg\" (UID: \"a582a43b-f0c6-40d4-8550-80a7f1dba309\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" Aug 01 00:35:47.135445 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:47.135382 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" Aug 01 00:35:47.267488 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:47.267452 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg"] Aug 01 00:35:47.270888 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:35:47.270859 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda582a43b_f0c6_40d4_8550_80a7f1dba309.slice/crio-dc7eae471f9781cc3fc125cb667bcceb6efda7919f7cfc6afa09d524a0888d61 WatchSource:0}: Error finding container dc7eae471f9781cc3fc125cb667bcceb6efda7919f7cfc6afa09d524a0888d61: Status 404 returned error can't find the container with id dc7eae471f9781cc3fc125cb667bcceb6efda7919f7cfc6afa09d524a0888d61 Aug 01 00:35:47.565444 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:35:47.565410 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" event={"ID":"a582a43b-f0c6-40d4-8550-80a7f1dba309","Type":"ContainerStarted","Data":"dc7eae471f9781cc3fc125cb667bcceb6efda7919f7cfc6afa09d524a0888d61"} Aug 01 00:36:03.646892 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:36:03.646850 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" event={"ID":"a582a43b-f0c6-40d4-8550-80a7f1dba309","Type":"ContainerStarted","Data":"ebac456974f0d627615bde6ce9f450e142168a0c289c758be8c32ded009d671b"} Aug 01 00:36:03.647276 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:36:03.646940 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" Aug 01 00:36:03.669822 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:36:03.669779 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" podStartSLOduration=1.424086582 podStartE2EDuration="17.669765216s" totalImagesPullingTime="16.245678634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:36:03.666982203 +0000 UTC m=+452.455704028" watchObservedRunningTime="2026-08-01 00:36:03.669765216 +0000 UTC m=+452.458487042" Aug 01 00:36:14.653457 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:36:14.653426 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" Aug 01 00:37:15.116975 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.116937 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f44b49488-vgvjn"] Aug 01 00:37:15.120281 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.120248 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44b49488-vgvjn" Aug 01 00:37:15.123733 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.123707 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 01 00:37:15.123733 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.123728 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 01 00:37:15.123913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.123706 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dgftb\"" Aug 01 00:37:15.128653 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.128630 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f44b49488-vgvjn"] Aug 01 00:37:15.241654 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.241624 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5e28501-848b-4eb4-aabf-1ebecb1848ea-webhook-cert\") pod \"maas-controller-f44b49488-vgvjn\" (UID: \"b5e28501-848b-4eb4-aabf-1ebecb1848ea\") " pod="opendatahub/maas-controller-f44b49488-vgvjn" Aug 01 00:37:15.241654 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.241660 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thjw7\" (UniqueName: \"kubernetes.io/projected/b5e28501-848b-4eb4-aabf-1ebecb1848ea-kube-api-access-thjw7\") pod \"maas-controller-f44b49488-vgvjn\" (UID: \"b5e28501-848b-4eb4-aabf-1ebecb1848ea\") " pod="opendatahub/maas-controller-f44b49488-vgvjn" Aug 01 00:37:15.342367 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.342341 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5e28501-848b-4eb4-aabf-1ebecb1848ea-webhook-cert\") pod \"maas-controller-f44b49488-vgvjn\" (UID: \"b5e28501-848b-4eb4-aabf-1ebecb1848ea\") " pod="opendatahub/maas-controller-f44b49488-vgvjn" Aug 01 00:37:15.342519 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.342380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thjw7\" (UniqueName: \"kubernetes.io/projected/b5e28501-848b-4eb4-aabf-1ebecb1848ea-kube-api-access-thjw7\") pod \"maas-controller-f44b49488-vgvjn\" (UID: \"b5e28501-848b-4eb4-aabf-1ebecb1848ea\") " pod="opendatahub/maas-controller-f44b49488-vgvjn" Aug 01 00:37:15.344657 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.344638 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5e28501-848b-4eb4-aabf-1ebecb1848ea-webhook-cert\") pod \"maas-controller-f44b49488-vgvjn\" (UID: \"b5e28501-848b-4eb4-aabf-1ebecb1848ea\") " pod="opendatahub/maas-controller-f44b49488-vgvjn" Aug 01 00:37:15.350144 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.350121 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thjw7\" (UniqueName: \"kubernetes.io/projected/b5e28501-848b-4eb4-aabf-1ebecb1848ea-kube-api-access-thjw7\") pod \"maas-controller-f44b49488-vgvjn\" (UID: \"b5e28501-848b-4eb4-aabf-1ebecb1848ea\") " pod="opendatahub/maas-controller-f44b49488-vgvjn" Aug 01 00:37:15.430712 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.430640 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44b49488-vgvjn" Aug 01 00:37:15.555932 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.555910 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f44b49488-vgvjn"] Aug 01 00:37:15.558517 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:37:15.558488 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e28501_848b_4eb4_aabf_1ebecb1848ea.slice/crio-6f2783929988437e8a4055fc4b02d052ca0ae22341159fa2aab998207cfce168 WatchSource:0}: Error finding container 6f2783929988437e8a4055fc4b02d052ca0ae22341159fa2aab998207cfce168: Status 404 returned error can't find the container with id 6f2783929988437e8a4055fc4b02d052ca0ae22341159fa2aab998207cfce168 Aug 01 00:37:15.964220 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:15.964188 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44b49488-vgvjn" event={"ID":"b5e28501-848b-4eb4-aabf-1ebecb1848ea","Type":"ContainerStarted","Data":"6f2783929988437e8a4055fc4b02d052ca0ae22341159fa2aab998207cfce168"} Aug 01 00:37:16.975185 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:16.973678 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44b49488-vgvjn" event={"ID":"b5e28501-848b-4eb4-aabf-1ebecb1848ea","Type":"ContainerStarted","Data":"f439cea994d06681baf8d0233302f0909e6e7a9367aa77238874d7af2605425a"} Aug 01 00:37:16.975185 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:16.973768 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f44b49488-vgvjn" Aug 01 00:37:16.995762 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:16.995703 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f44b49488-vgvjn" podStartSLOduration=0.693594113 podStartE2EDuration="1.995687725s" totalImagesPullingTime="1.302093612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:16.994821342 +0000 UTC m=+525.783543180" watchObservedRunningTime="2026-08-01 00:37:16.995687725 +0000 UTC m=+525.784409833" Aug 01 00:37:27.980913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:27.980831 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f44b49488-vgvjn" Aug 01 00:37:50.732865 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:50.732828 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s"] Aug 01 00:37:50.736251 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:50.736232 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s" Aug 01 00:37:50.738464 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:50.738446 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Aug 01 00:37:50.744861 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:50.744835 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s"] Aug 01 00:37:50.758859 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:50.758822 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq98d\" (UniqueName: \"kubernetes.io/projected/0568d048-0ee5-4e6d-a26e-416732ccdc02-kube-api-access-tq98d\") pod \"payload-pre-processing-687dbb6cf6-9bm8s\" (UID: \"0568d048-0ee5-4e6d-a26e-416732ccdc02\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s" Aug 01 00:37:50.758971 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:50.758884 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0568d048-0ee5-4e6d-a26e-416732ccdc02-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-9bm8s\" (UID: \"0568d048-0ee5-4e6d-a26e-416732ccdc02\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s" Aug 01 00:37:50.859841 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:50.859805 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq98d\" (UniqueName: \"kubernetes.io/projected/0568d048-0ee5-4e6d-a26e-416732ccdc02-kube-api-access-tq98d\") pod \"payload-pre-processing-687dbb6cf6-9bm8s\" (UID: \"0568d048-0ee5-4e6d-a26e-416732ccdc02\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s" Aug 01 00:37:50.860101 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:50.859859 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0568d048-0ee5-4e6d-a26e-416732ccdc02-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-9bm8s\" (UID: \"0568d048-0ee5-4e6d-a26e-416732ccdc02\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s" Aug 01 00:37:50.860703 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:50.860676 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0568d048-0ee5-4e6d-a26e-416732ccdc02-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-9bm8s\" (UID: \"0568d048-0ee5-4e6d-a26e-416732ccdc02\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s" Aug 01 00:37:50.867751 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:50.867728 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq98d\" (UniqueName: \"kubernetes.io/projected/0568d048-0ee5-4e6d-a26e-416732ccdc02-kube-api-access-tq98d\") pod \"payload-pre-processing-687dbb6cf6-9bm8s\" (UID: \"0568d048-0ee5-4e6d-a26e-416732ccdc02\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s" Aug 01 00:37:51.047633 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:51.047556 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s" Aug 01 00:37:51.183547 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:51.183460 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s"] Aug 01 00:37:51.185924 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:37:51.185889 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0568d048_0ee5_4e6d_a26e_416732ccdc02.slice/crio-78bb9a6f4051aa90e0383456a608180a0a0d75e986be1f145bcede46c20d4e46 WatchSource:0}: Error finding container 78bb9a6f4051aa90e0383456a608180a0a0d75e986be1f145bcede46c20d4e46: Status 404 returned error can't find the container with id 78bb9a6f4051aa90e0383456a608180a0a0d75e986be1f145bcede46c20d4e46 Aug 01 00:37:52.129141 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:52.129103 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s" event={"ID":"0568d048-0ee5-4e6d-a26e-416732ccdc02","Type":"ContainerStarted","Data":"78bb9a6f4051aa90e0383456a608180a0a0d75e986be1f145bcede46c20d4e46"} Aug 01 00:37:53.134730 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:53.134639 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s" event={"ID":"0568d048-0ee5-4e6d-a26e-416732ccdc02","Type":"ContainerStarted","Data":"e11c2f9e088f9423a41c57ba274a6c4d0d1dc60e3441aac3ff1000ff03a270ca"} Aug 01 00:37:53.135213 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:53.134784 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s" Aug 01 00:37:53.150768 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:37:53.150720 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s" podStartSLOduration=1.498096648 podStartE2EDuration="3.150705863s" totalImagesPullingTime="1.652609215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:53.150126958 +0000 UTC m=+561.938848801" watchObservedRunningTime="2026-08-01 00:37:53.150705863 +0000 UTC m=+561.939427753" Aug 01 00:38:04.142104 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:04.142051 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9bm8s" Aug 01 00:38:09.539268 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.539238 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4"] Aug 01 00:38:09.542516 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.542494 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.545992 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.545749 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 01 00:38:09.546206 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.546179 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 01 00:38:09.546316 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.546273 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Aug 01 00:38:09.546419 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.546396 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fzrd7\"" Aug 01 00:38:09.553650 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.553632 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4"] Aug 01 00:38:09.638111 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.638080 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.638257 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.638114 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.638257 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.638183 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.638334 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.638261 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.638334 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.638294 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8j9\" (UniqueName: \"kubernetes.io/projected/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-kube-api-access-zn8j9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.638334 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.638322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.738918 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.738885 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.738918 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.738922 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8j9\" (UniqueName: \"kubernetes.io/projected/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-kube-api-access-zn8j9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.739187 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.738947 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.739187 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.738997 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.739187 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.739016 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.739187 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.739171 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.739462 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.739433 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.739587 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.739500 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.739587 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.739535 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.741211 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.741191 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.741464 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.741445 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.746344 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.746324 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8j9\" (UniqueName: \"kubernetes.io/projected/37c3dc90-dd65-45f4-9ff4-ed81c7baeba1-kube-api-access-zn8j9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4\" (UID: \"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.854712 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.854691 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:09.987189 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:09.987066 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4"] Aug 01 00:38:09.989525 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:38:09.989498 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c3dc90_dd65_45f4_9ff4_ed81c7baeba1.slice/crio-d380d8257e3011f7a9b8e37843cb1056e9260d8e8524572921a678bf770a1267 WatchSource:0}: Error finding container d380d8257e3011f7a9b8e37843cb1056e9260d8e8524572921a678bf770a1267: Status 404 returned error can't find the container with id d380d8257e3011f7a9b8e37843cb1056e9260d8e8524572921a678bf770a1267 Aug 01 00:38:10.210514 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:10.210441 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" event={"ID":"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1","Type":"ContainerStarted","Data":"d380d8257e3011f7a9b8e37843cb1056e9260d8e8524572921a678bf770a1267"} Aug 01 00:38:11.218158 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:11.218089 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" event={"ID":"37c3dc90-dd65-45f4-9ff4-ed81c7baeba1","Type":"ContainerStarted","Data":"b33ebfac425788d4a33c76cb504ef744004c8550a64f4e1c3ea18de3bb40cb35"} Aug 01 00:38:11.238150 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:11.238104 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" podStartSLOduration=1.087814297 podStartE2EDuration="2.238088558s" totalImagesPullingTime="1.150274261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:11.235835058 +0000 UTC m=+580.024556885" watchObservedRunningTime="2026-08-01 00:38:11.238088558 +0000 UTC m=+580.026810378" Aug 01 00:38:19.855282 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:19.855253 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:19.855629 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:19.855310 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:19.867464 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:19.867441 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:20.267035 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:20.266957 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4" Aug 01 00:38:31.659496 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:31.659470 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:38:31.661183 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:31.661161 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:38:31.665068 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:31.665034 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:38:31.666545 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:38:31.666527 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:39:08.536857 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.536783 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz"] Aug 01 00:39:08.540295 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.540279 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.543198 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.543175 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Aug 01 00:39:08.552043 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.552018 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz"] Aug 01 00:39:08.617545 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.617502 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdda65f4-0d04-44ec-9f42-5dc43126a777-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.617708 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.617580 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdda65f4-0d04-44ec-9f42-5dc43126a777-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.617708 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.617604 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdda65f4-0d04-44ec-9f42-5dc43126a777-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.617708 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.617622 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdda65f4-0d04-44ec-9f42-5dc43126a777-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.617867 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.617721 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdda65f4-0d04-44ec-9f42-5dc43126a777-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.617867 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.617758 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55lvh\" (UniqueName: \"kubernetes.io/projected/fdda65f4-0d04-44ec-9f42-5dc43126a777-kube-api-access-55lvh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.718408 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.718367 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdda65f4-0d04-44ec-9f42-5dc43126a777-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.718632 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.718414 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55lvh\" (UniqueName: \"kubernetes.io/projected/fdda65f4-0d04-44ec-9f42-5dc43126a777-kube-api-access-55lvh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.718632 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.718481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdda65f4-0d04-44ec-9f42-5dc43126a777-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.718632 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.718528 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdda65f4-0d04-44ec-9f42-5dc43126a777-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.718632 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.718559 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdda65f4-0d04-44ec-9f42-5dc43126a777-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.718632 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.718587 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdda65f4-0d04-44ec-9f42-5dc43126a777-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.719035 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.718952 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdda65f4-0d04-44ec-9f42-5dc43126a777-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.719169 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.719144 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdda65f4-0d04-44ec-9f42-5dc43126a777-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.719256 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.719168 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdda65f4-0d04-44ec-9f42-5dc43126a777-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.721233 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.721207 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdda65f4-0d04-44ec-9f42-5dc43126a777-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.721709 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.721694 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdda65f4-0d04-44ec-9f42-5dc43126a777-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.727371 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.727320 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55lvh\" (UniqueName: \"kubernetes.io/projected/fdda65f4-0d04-44ec-9f42-5dc43126a777-kube-api-access-55lvh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4l7vz\" (UID: \"fdda65f4-0d04-44ec-9f42-5dc43126a777\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.852551 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.852522 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:08.980485 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.980462 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz"] Aug 01 00:39:08.982408 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:39:08.982378 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdda65f4_0d04_44ec_9f42_5dc43126a777.slice/crio-7c10357b6daadf2db84dbd177440f985ea7b0c0dfc309df1f13f5c24843501d3 WatchSource:0}: Error finding container 7c10357b6daadf2db84dbd177440f985ea7b0c0dfc309df1f13f5c24843501d3: Status 404 returned error can't find the container with id 7c10357b6daadf2db84dbd177440f985ea7b0c0dfc309df1f13f5c24843501d3 Aug 01 00:39:08.983919 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:08.983897 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 00:39:09.476637 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:09.476555 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" event={"ID":"fdda65f4-0d04-44ec-9f42-5dc43126a777","Type":"ContainerStarted","Data":"d56f9249b669749b69dfac6c68d23e991f7e7cec5593e97246dc5243abe3c839"} Aug 01 00:39:09.476775 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:09.476639 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" event={"ID":"fdda65f4-0d04-44ec-9f42-5dc43126a777","Type":"ContainerStarted","Data":"7c10357b6daadf2db84dbd177440f985ea7b0c0dfc309df1f13f5c24843501d3"} Aug 01 00:39:09.495048 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:09.495005 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" podStartSLOduration=1.292819999 podStartE2EDuration="1.494992797s" totalImagesPullingTime="202.172798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:39:09.493804263 +0000 UTC m=+638.282526091" watchObservedRunningTime="2026-08-01 00:39:09.494992797 +0000 UTC m=+638.283714622" Aug 01 00:39:18.853375 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:18.853340 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:18.853751 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:18.853398 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:18.865762 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:18.865734 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:39:19.534885 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:39:19.534859 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4l7vz" Aug 01 00:40:55.048930 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.048894 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f44b49488-vgvjn"] Aug 01 00:40:55.049554 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.049150 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f44b49488-vgvjn" podUID="b5e28501-848b-4eb4-aabf-1ebecb1848ea" containerName="manager" containerID="cri-o://f439cea994d06681baf8d0233302f0909e6e7a9367aa77238874d7af2605425a" gracePeriod=30 Aug 01 00:40:55.298906 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.298883 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44b49488-vgvjn" Aug 01 00:40:55.436215 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.436184 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thjw7\" (UniqueName: \"kubernetes.io/projected/b5e28501-848b-4eb4-aabf-1ebecb1848ea-kube-api-access-thjw7\") pod \"b5e28501-848b-4eb4-aabf-1ebecb1848ea\" (UID: \"b5e28501-848b-4eb4-aabf-1ebecb1848ea\") " Aug 01 00:40:55.436376 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.436250 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5e28501-848b-4eb4-aabf-1ebecb1848ea-webhook-cert\") pod \"b5e28501-848b-4eb4-aabf-1ebecb1848ea\" (UID: \"b5e28501-848b-4eb4-aabf-1ebecb1848ea\") " Aug 01 00:40:55.438234 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.438198 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e28501-848b-4eb4-aabf-1ebecb1848ea-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b5e28501-848b-4eb4-aabf-1ebecb1848ea" (UID: "b5e28501-848b-4eb4-aabf-1ebecb1848ea"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:40:55.438334 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.438282 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e28501-848b-4eb4-aabf-1ebecb1848ea-kube-api-access-thjw7" (OuterVolumeSpecName: "kube-api-access-thjw7") pod "b5e28501-848b-4eb4-aabf-1ebecb1848ea" (UID: "b5e28501-848b-4eb4-aabf-1ebecb1848ea"). InnerVolumeSpecName "kube-api-access-thjw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:40:55.537660 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.537632 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5e28501-848b-4eb4-aabf-1ebecb1848ea-webhook-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:40:55.537660 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.537660 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thjw7\" (UniqueName: \"kubernetes.io/projected/b5e28501-848b-4eb4-aabf-1ebecb1848ea-kube-api-access-thjw7\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:40:55.957140 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.957103 2563 generic.go:356] "Generic (PLEG): container finished" podID="b5e28501-848b-4eb4-aabf-1ebecb1848ea" containerID="f439cea994d06681baf8d0233302f0909e6e7a9367aa77238874d7af2605425a" exitCode=0 Aug 01 00:40:55.957295 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.957149 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44b49488-vgvjn" Aug 01 00:40:55.957295 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.957190 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44b49488-vgvjn" event={"ID":"b5e28501-848b-4eb4-aabf-1ebecb1848ea","Type":"ContainerDied","Data":"f439cea994d06681baf8d0233302f0909e6e7a9367aa77238874d7af2605425a"} Aug 01 00:40:55.957295 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.957247 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44b49488-vgvjn" event={"ID":"b5e28501-848b-4eb4-aabf-1ebecb1848ea","Type":"ContainerDied","Data":"6f2783929988437e8a4055fc4b02d052ca0ae22341159fa2aab998207cfce168"} Aug 01 00:40:55.957295 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.957277 2563 scope.go:117] "RemoveContainer" containerID="f439cea994d06681baf8d0233302f0909e6e7a9367aa77238874d7af2605425a" Aug 01 00:40:55.965824 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.965808 2563 scope.go:117] "RemoveContainer" containerID="f439cea994d06681baf8d0233302f0909e6e7a9367aa77238874d7af2605425a" Aug 01 00:40:55.966102 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:40:55.966071 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f439cea994d06681baf8d0233302f0909e6e7a9367aa77238874d7af2605425a\": container with ID starting with f439cea994d06681baf8d0233302f0909e6e7a9367aa77238874d7af2605425a not found: ID does not exist" containerID="f439cea994d06681baf8d0233302f0909e6e7a9367aa77238874d7af2605425a" Aug 01 00:40:55.966176 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.966112 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f439cea994d06681baf8d0233302f0909e6e7a9367aa77238874d7af2605425a"} err="failed to get container status \"f439cea994d06681baf8d0233302f0909e6e7a9367aa77238874d7af2605425a\": rpc error: code = NotFound desc = could not find container \"f439cea994d06681baf8d0233302f0909e6e7a9367aa77238874d7af2605425a\": container with ID starting with f439cea994d06681baf8d0233302f0909e6e7a9367aa77238874d7af2605425a not found: ID does not exist" Aug 01 00:40:55.974461 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.974444 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f44b49488-vgvjn"] Aug 01 00:40:55.979444 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:55.979424 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f44b49488-vgvjn"] Aug 01 00:40:57.733044 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:40:57.733002 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5e28501-848b-4eb4-aabf-1ebecb1848ea" path="/var/lib/kubelet/pods/b5e28501-848b-4eb4-aabf-1ebecb1848ea/volumes" Aug 01 00:41:56.822594 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.822508 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f44b49488-cmqgn"] Aug 01 00:41:56.823873 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.823843 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5e28501-848b-4eb4-aabf-1ebecb1848ea" containerName="manager" Aug 01 00:41:56.824004 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.823994 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5e28501-848b-4eb4-aabf-1ebecb1848ea" containerName="manager" Aug 01 00:41:56.824269 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.824252 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5e28501-848b-4eb4-aabf-1ebecb1848ea" containerName="manager" Aug 01 00:41:56.829417 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.829393 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44b49488-cmqgn" Aug 01 00:41:56.834212 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.834189 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 01 00:41:56.834674 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.834644 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dgftb\"" Aug 01 00:41:56.834984 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.834962 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f44b49488-cmqgn"] Aug 01 00:41:56.835332 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.835315 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 01 00:41:56.856851 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.856830 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5dpx\" (UniqueName: \"kubernetes.io/projected/d95778b0-b8d6-4234-be14-53a6c9038d00-kube-api-access-b5dpx\") pod \"maas-controller-f44b49488-cmqgn\" (UID: \"d95778b0-b8d6-4234-be14-53a6c9038d00\") " pod="opendatahub/maas-controller-f44b49488-cmqgn" Aug 01 00:41:56.856955 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.856895 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d95778b0-b8d6-4234-be14-53a6c9038d00-webhook-cert\") pod \"maas-controller-f44b49488-cmqgn\" (UID: \"d95778b0-b8d6-4234-be14-53a6c9038d00\") " pod="opendatahub/maas-controller-f44b49488-cmqgn" Aug 01 00:41:56.957817 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.957780 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d95778b0-b8d6-4234-be14-53a6c9038d00-webhook-cert\") pod \"maas-controller-f44b49488-cmqgn\" (UID: \"d95778b0-b8d6-4234-be14-53a6c9038d00\") " pod="opendatahub/maas-controller-f44b49488-cmqgn" Aug 01 00:41:56.957963 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.957846 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5dpx\" (UniqueName: \"kubernetes.io/projected/d95778b0-b8d6-4234-be14-53a6c9038d00-kube-api-access-b5dpx\") pod \"maas-controller-f44b49488-cmqgn\" (UID: \"d95778b0-b8d6-4234-be14-53a6c9038d00\") " pod="opendatahub/maas-controller-f44b49488-cmqgn" Aug 01 00:41:56.960198 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.960174 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d95778b0-b8d6-4234-be14-53a6c9038d00-webhook-cert\") pod \"maas-controller-f44b49488-cmqgn\" (UID: \"d95778b0-b8d6-4234-be14-53a6c9038d00\") " pod="opendatahub/maas-controller-f44b49488-cmqgn" Aug 01 00:41:56.967814 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:56.967791 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5dpx\" (UniqueName: \"kubernetes.io/projected/d95778b0-b8d6-4234-be14-53a6c9038d00-kube-api-access-b5dpx\") pod \"maas-controller-f44b49488-cmqgn\" (UID: \"d95778b0-b8d6-4234-be14-53a6c9038d00\") " pod="opendatahub/maas-controller-f44b49488-cmqgn" Aug 01 00:41:57.141718 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:57.141623 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44b49488-cmqgn" Aug 01 00:41:57.267560 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:57.267537 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f44b49488-cmqgn"] Aug 01 00:41:57.269705 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:41:57.269674 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95778b0_b8d6_4234_be14_53a6c9038d00.slice/crio-3fb2e814d6eb88af793736e320eaefd6e87f5eff5518c1e5c4d6b6a5d9dbf547 WatchSource:0}: Error finding container 3fb2e814d6eb88af793736e320eaefd6e87f5eff5518c1e5c4d6b6a5d9dbf547: Status 404 returned error can't find the container with id 3fb2e814d6eb88af793736e320eaefd6e87f5eff5518c1e5c4d6b6a5d9dbf547 Aug 01 00:41:58.233928 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:58.233842 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44b49488-cmqgn" event={"ID":"d95778b0-b8d6-4234-be14-53a6c9038d00","Type":"ContainerStarted","Data":"a3c53aee5d6f5144358ec708da4d00e6777292c9d9ee877881e93cd89b30d8e1"} Aug 01 00:41:58.233928 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:58.233892 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44b49488-cmqgn" event={"ID":"d95778b0-b8d6-4234-be14-53a6c9038d00","Type":"ContainerStarted","Data":"3fb2e814d6eb88af793736e320eaefd6e87f5eff5518c1e5c4d6b6a5d9dbf547"} Aug 01 00:41:58.234351 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:58.233934 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f44b49488-cmqgn" Aug 01 00:41:58.253912 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:41:58.253857 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f44b49488-cmqgn" podStartSLOduration=1.630647527 podStartE2EDuration="2.253842352s" totalImagesPullingTime="623.194825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:41:58.253013556 +0000 UTC m=+807.041735404" watchObservedRunningTime="2026-08-01 00:41:58.253842352 +0000 UTC m=+807.042564177" Aug 01 00:42:09.240842 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:09.240814 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f44b49488-cmqgn" Aug 01 00:42:13.872031 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:13.871996 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f44b49488-cmqgn"] Aug 01 00:42:13.872418 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:13.872250 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f44b49488-cmqgn" podUID="d95778b0-b8d6-4234-be14-53a6c9038d00" containerName="manager" containerID="cri-o://a3c53aee5d6f5144358ec708da4d00e6777292c9d9ee877881e93cd89b30d8e1" gracePeriod=30 Aug 01 00:42:14.117902 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.117878 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44b49488-cmqgn" Aug 01 00:42:14.191194 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.191105 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5dpx\" (UniqueName: \"kubernetes.io/projected/d95778b0-b8d6-4234-be14-53a6c9038d00-kube-api-access-b5dpx\") pod \"d95778b0-b8d6-4234-be14-53a6c9038d00\" (UID: \"d95778b0-b8d6-4234-be14-53a6c9038d00\") " Aug 01 00:42:14.191194 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.191140 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d95778b0-b8d6-4234-be14-53a6c9038d00-webhook-cert\") pod \"d95778b0-b8d6-4234-be14-53a6c9038d00\" (UID: \"d95778b0-b8d6-4234-be14-53a6c9038d00\") " Aug 01 00:42:14.193194 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.193167 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95778b0-b8d6-4234-be14-53a6c9038d00-kube-api-access-b5dpx" (OuterVolumeSpecName: "kube-api-access-b5dpx") pod "d95778b0-b8d6-4234-be14-53a6c9038d00" (UID: "d95778b0-b8d6-4234-be14-53a6c9038d00"). InnerVolumeSpecName "kube-api-access-b5dpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:42:14.193273 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.193212 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95778b0-b8d6-4234-be14-53a6c9038d00-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d95778b0-b8d6-4234-be14-53a6c9038d00" (UID: "d95778b0-b8d6-4234-be14-53a6c9038d00"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:42:14.292807 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.292768 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5dpx\" (UniqueName: \"kubernetes.io/projected/d95778b0-b8d6-4234-be14-53a6c9038d00-kube-api-access-b5dpx\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:42:14.292807 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.292803 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d95778b0-b8d6-4234-be14-53a6c9038d00-webhook-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:42:14.306826 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.306793 2563 generic.go:356] "Generic (PLEG): container finished" podID="d95778b0-b8d6-4234-be14-53a6c9038d00" containerID="a3c53aee5d6f5144358ec708da4d00e6777292c9d9ee877881e93cd89b30d8e1" exitCode=0 Aug 01 00:42:14.307012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.306837 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44b49488-cmqgn" Aug 01 00:42:14.307012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.306879 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44b49488-cmqgn" event={"ID":"d95778b0-b8d6-4234-be14-53a6c9038d00","Type":"ContainerDied","Data":"a3c53aee5d6f5144358ec708da4d00e6777292c9d9ee877881e93cd89b30d8e1"} Aug 01 00:42:14.307012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.306932 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44b49488-cmqgn" event={"ID":"d95778b0-b8d6-4234-be14-53a6c9038d00","Type":"ContainerDied","Data":"3fb2e814d6eb88af793736e320eaefd6e87f5eff5518c1e5c4d6b6a5d9dbf547"} Aug 01 00:42:14.307012 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.306962 2563 scope.go:117] "RemoveContainer" containerID="a3c53aee5d6f5144358ec708da4d00e6777292c9d9ee877881e93cd89b30d8e1" Aug 01 00:42:14.316331 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.316313 2563 scope.go:117] "RemoveContainer" containerID="a3c53aee5d6f5144358ec708da4d00e6777292c9d9ee877881e93cd89b30d8e1" Aug 01 00:42:14.316600 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:42:14.316581 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c53aee5d6f5144358ec708da4d00e6777292c9d9ee877881e93cd89b30d8e1\": container with ID starting with a3c53aee5d6f5144358ec708da4d00e6777292c9d9ee877881e93cd89b30d8e1 not found: ID does not exist" containerID="a3c53aee5d6f5144358ec708da4d00e6777292c9d9ee877881e93cd89b30d8e1" Aug 01 00:42:14.316645 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.316608 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c53aee5d6f5144358ec708da4d00e6777292c9d9ee877881e93cd89b30d8e1"} err="failed to get container status \"a3c53aee5d6f5144358ec708da4d00e6777292c9d9ee877881e93cd89b30d8e1\": rpc error: code = NotFound desc = could not find container \"a3c53aee5d6f5144358ec708da4d00e6777292c9d9ee877881e93cd89b30d8e1\": container with ID starting with a3c53aee5d6f5144358ec708da4d00e6777292c9d9ee877881e93cd89b30d8e1 not found: ID does not exist" Aug 01 00:42:14.335345 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.335318 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f44b49488-cmqgn"] Aug 01 00:42:14.340454 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:14.340423 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f44b49488-cmqgn"] Aug 01 00:42:15.112375 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.112338 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f44b49488-lbqlf"] Aug 01 00:42:15.112819 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.112673 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95778b0-b8d6-4234-be14-53a6c9038d00" containerName="manager" Aug 01 00:42:15.112819 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.112685 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95778b0-b8d6-4234-be14-53a6c9038d00" containerName="manager" Aug 01 00:42:15.112819 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.112758 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95778b0-b8d6-4234-be14-53a6c9038d00" containerName="manager" Aug 01 00:42:15.116870 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.116852 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44b49488-lbqlf" Aug 01 00:42:15.121436 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.121406 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 01 00:42:15.122516 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.122493 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 01 00:42:15.122599 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.122542 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dgftb\"" Aug 01 00:42:15.127181 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.127156 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f44b49488-lbqlf"] Aug 01 00:42:15.202627 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.202590 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92n66\" (UniqueName: \"kubernetes.io/projected/80beabaa-abdf-414b-8869-f6e808e11faa-kube-api-access-92n66\") pod \"maas-controller-f44b49488-lbqlf\" (UID: \"80beabaa-abdf-414b-8869-f6e808e11faa\") " pod="opendatahub/maas-controller-f44b49488-lbqlf" Aug 01 00:42:15.202814 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.202636 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80beabaa-abdf-414b-8869-f6e808e11faa-webhook-cert\") pod \"maas-controller-f44b49488-lbqlf\" (UID: \"80beabaa-abdf-414b-8869-f6e808e11faa\") " pod="opendatahub/maas-controller-f44b49488-lbqlf" Aug 01 00:42:15.304151 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.304118 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92n66\" (UniqueName: \"kubernetes.io/projected/80beabaa-abdf-414b-8869-f6e808e11faa-kube-api-access-92n66\") pod \"maas-controller-f44b49488-lbqlf\" (UID: \"80beabaa-abdf-414b-8869-f6e808e11faa\") " pod="opendatahub/maas-controller-f44b49488-lbqlf" Aug 01 00:42:15.304318 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.304159 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80beabaa-abdf-414b-8869-f6e808e11faa-webhook-cert\") pod \"maas-controller-f44b49488-lbqlf\" (UID: \"80beabaa-abdf-414b-8869-f6e808e11faa\") " pod="opendatahub/maas-controller-f44b49488-lbqlf" Aug 01 00:42:15.306518 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.306496 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80beabaa-abdf-414b-8869-f6e808e11faa-webhook-cert\") pod \"maas-controller-f44b49488-lbqlf\" (UID: \"80beabaa-abdf-414b-8869-f6e808e11faa\") " pod="opendatahub/maas-controller-f44b49488-lbqlf" Aug 01 00:42:15.315119 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.315094 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92n66\" (UniqueName: \"kubernetes.io/projected/80beabaa-abdf-414b-8869-f6e808e11faa-kube-api-access-92n66\") pod \"maas-controller-f44b49488-lbqlf\" (UID: \"80beabaa-abdf-414b-8869-f6e808e11faa\") " pod="opendatahub/maas-controller-f44b49488-lbqlf" Aug 01 00:42:15.427264 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.427172 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44b49488-lbqlf" Aug 01 00:42:15.558617 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.558591 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f44b49488-lbqlf"] Aug 01 00:42:15.560648 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:42:15.560617 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80beabaa_abdf_414b_8869_f6e808e11faa.slice/crio-23fe50555df600c0269996981a2b0eff4a1382df3613241a3aa407d47c0e7479 WatchSource:0}: Error finding container 23fe50555df600c0269996981a2b0eff4a1382df3613241a3aa407d47c0e7479: Status 404 returned error can't find the container with id 23fe50555df600c0269996981a2b0eff4a1382df3613241a3aa407d47c0e7479 Aug 01 00:42:15.728140 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:15.728043 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d95778b0-b8d6-4234-be14-53a6c9038d00" path="/var/lib/kubelet/pods/d95778b0-b8d6-4234-be14-53a6c9038d00/volumes" Aug 01 00:42:16.318582 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:16.318555 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44b49488-lbqlf" event={"ID":"80beabaa-abdf-414b-8869-f6e808e11faa","Type":"ContainerStarted","Data":"83b06b78877c95e07870cedc484559c214eb87f5c61a91272028e7641b0c9881"} Aug 01 00:42:16.319015 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:16.318612 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44b49488-lbqlf" event={"ID":"80beabaa-abdf-414b-8869-f6e808e11faa","Type":"ContainerStarted","Data":"23fe50555df600c0269996981a2b0eff4a1382df3613241a3aa407d47c0e7479"} Aug 01 00:42:16.319015 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:16.318661 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f44b49488-lbqlf" Aug 01 00:42:16.338755 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:16.338703 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f44b49488-lbqlf" podStartSLOduration=0.919793372 podStartE2EDuration="1.338688564s" totalImagesPullingTime="418.895192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:42:16.336978946 +0000 UTC m=+825.125700773" watchObservedRunningTime="2026-08-01 00:42:16.338688564 +0000 UTC m=+825.127410413" Aug 01 00:42:27.326649 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:42:27.326617 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f44b49488-lbqlf" Aug 01 00:43:31.689639 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:43:31.689562 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:43:31.690156 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:43:31.689930 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:43:31.696836 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:43:31.696815 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:43:31.697028 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:43:31.697008 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:45:00.142922 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:00.142824 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759085-bvnz8"] Aug 01 00:45:00.146264 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:00.146249 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759085-bvnz8" Aug 01 00:45:00.148699 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:00.148682 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68mnr\"" Aug 01 00:45:00.155463 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:00.155444 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759085-bvnz8" Aug 01 00:45:00.159860 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:00.159828 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759085-bvnz8"] Aug 01 00:45:00.500439 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:00.500415 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759085-bvnz8"] Aug 01 00:45:00.502803 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:45:00.502771 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc787ccd8_b889_48ec_afe2_3c900e24c4e9.slice/crio-3c7fbf1f61c2a9138c5dc89ea1a562019cd03740c501293e0007fea03c14f1c4 WatchSource:0}: Error finding container 3c7fbf1f61c2a9138c5dc89ea1a562019cd03740c501293e0007fea03c14f1c4: Status 404 returned error can't find the container with id 3c7fbf1f61c2a9138c5dc89ea1a562019cd03740c501293e0007fea03c14f1c4 Aug 01 00:45:00.504671 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:00.504656 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 00:45:01.072620 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:01.072586 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759085-bvnz8" event={"ID":"c787ccd8-b889-48ec-afe2-3c900e24c4e9","Type":"ContainerStarted","Data":"3c7fbf1f61c2a9138c5dc89ea1a562019cd03740c501293e0007fea03c14f1c4"} Aug 01 00:45:05.094194 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:05.094157 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759085-bvnz8" event={"ID":"c787ccd8-b889-48ec-afe2-3c900e24c4e9","Type":"ContainerStarted","Data":"53d598fab93e7cdee6c1d624024f84a69ed8f26788a034635cf7b48796bd874f"} Aug 01 00:45:05.109139 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:05.109095 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759085-bvnz8" podStartSLOduration=1.384429769 podStartE2EDuration="5.109080502s" totalImagesPullingTime="3.724650733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:45:05.10822682 +0000 UTC m=+993.896948648" watchObservedRunningTime="2026-08-01 00:45:05.109080502 +0000 UTC m=+993.897802349" Aug 01 00:45:06.100006 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:06.099975 2563 generic.go:356] "Generic (PLEG): container finished" podID="c787ccd8-b889-48ec-afe2-3c900e24c4e9" containerID="53d598fab93e7cdee6c1d624024f84a69ed8f26788a034635cf7b48796bd874f" exitCode=0 Aug 01 00:45:06.100391 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:06.100032 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759085-bvnz8" event={"ID":"c787ccd8-b889-48ec-afe2-3c900e24c4e9","Type":"ContainerDied","Data":"53d598fab93e7cdee6c1d624024f84a69ed8f26788a034635cf7b48796bd874f"} Aug 01 00:45:07.232863 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:07.232840 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759085-bvnz8" Aug 01 00:45:08.110206 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:08.110169 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759085-bvnz8" event={"ID":"c787ccd8-b889-48ec-afe2-3c900e24c4e9","Type":"ContainerDied","Data":"3c7fbf1f61c2a9138c5dc89ea1a562019cd03740c501293e0007fea03c14f1c4"} Aug 01 00:45:08.110388 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:08.110226 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c7fbf1f61c2a9138c5dc89ea1a562019cd03740c501293e0007fea03c14f1c4" Aug 01 00:45:08.110388 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:45:08.110182 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759085-bvnz8" Aug 01 00:48:31.720216 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:31.720136 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:48:31.722746 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:31.722720 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:48:31.727135 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:31.727112 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:48:31.729437 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:31.729416 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:48:43.095701 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:43.095660 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg"] Aug 01 00:48:43.096229 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:43.095959 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" podUID="a582a43b-f0c6-40d4-8550-80a7f1dba309" containerName="manager" containerID="cri-o://ebac456974f0d627615bde6ce9f450e142168a0c289c758be8c32ded009d671b" gracePeriod=10 Aug 01 00:48:43.358072 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:43.357991 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" Aug 01 00:48:43.509497 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:43.509464 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a582a43b-f0c6-40d4-8550-80a7f1dba309-extensions-socket-volume\") pod \"a582a43b-f0c6-40d4-8550-80a7f1dba309\" (UID: \"a582a43b-f0c6-40d4-8550-80a7f1dba309\") " Aug 01 00:48:43.509666 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:43.509512 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kcsp\" (UniqueName: \"kubernetes.io/projected/a582a43b-f0c6-40d4-8550-80a7f1dba309-kube-api-access-2kcsp\") pod \"a582a43b-f0c6-40d4-8550-80a7f1dba309\" (UID: \"a582a43b-f0c6-40d4-8550-80a7f1dba309\") " Aug 01 00:48:43.509913 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:43.509884 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a582a43b-f0c6-40d4-8550-80a7f1dba309-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a582a43b-f0c6-40d4-8550-80a7f1dba309" (UID: "a582a43b-f0c6-40d4-8550-80a7f1dba309"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:48:43.511744 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:43.511720 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a582a43b-f0c6-40d4-8550-80a7f1dba309-kube-api-access-2kcsp" (OuterVolumeSpecName: "kube-api-access-2kcsp") pod "a582a43b-f0c6-40d4-8550-80a7f1dba309" (UID: "a582a43b-f0c6-40d4-8550-80a7f1dba309"). InnerVolumeSpecName "kube-api-access-2kcsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:48:43.610862 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:43.610772 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a582a43b-f0c6-40d4-8550-80a7f1dba309-extensions-socket-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:48:43.610862 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:43.610808 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kcsp\" (UniqueName: \"kubernetes.io/projected/a582a43b-f0c6-40d4-8550-80a7f1dba309-kube-api-access-2kcsp\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:48:44.112714 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:44.112674 2563 generic.go:356] "Generic (PLEG): container finished" podID="a582a43b-f0c6-40d4-8550-80a7f1dba309" containerID="ebac456974f0d627615bde6ce9f450e142168a0c289c758be8c32ded009d671b" exitCode=0 Aug 01 00:48:44.113127 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:44.112724 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" Aug 01 00:48:44.113127 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:44.112734 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" event={"ID":"a582a43b-f0c6-40d4-8550-80a7f1dba309","Type":"ContainerDied","Data":"ebac456974f0d627615bde6ce9f450e142168a0c289c758be8c32ded009d671b"} Aug 01 00:48:44.113127 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:44.112787 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg" event={"ID":"a582a43b-f0c6-40d4-8550-80a7f1dba309","Type":"ContainerDied","Data":"dc7eae471f9781cc3fc125cb667bcceb6efda7919f7cfc6afa09d524a0888d61"} Aug 01 00:48:44.113127 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:44.112816 2563 scope.go:117] "RemoveContainer" containerID="ebac456974f0d627615bde6ce9f450e142168a0c289c758be8c32ded009d671b" Aug 01 00:48:44.121852 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:44.121829 2563 scope.go:117] "RemoveContainer" containerID="ebac456974f0d627615bde6ce9f450e142168a0c289c758be8c32ded009d671b" Aug 01 00:48:44.122134 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:48:44.122113 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebac456974f0d627615bde6ce9f450e142168a0c289c758be8c32ded009d671b\": container with ID starting with ebac456974f0d627615bde6ce9f450e142168a0c289c758be8c32ded009d671b not found: ID does not exist" containerID="ebac456974f0d627615bde6ce9f450e142168a0c289c758be8c32ded009d671b" Aug 01 00:48:44.122206 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:44.122142 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebac456974f0d627615bde6ce9f450e142168a0c289c758be8c32ded009d671b"} err="failed to get container status \"ebac456974f0d627615bde6ce9f450e142168a0c289c758be8c32ded009d671b\": rpc error: code = NotFound desc = could not find container \"ebac456974f0d627615bde6ce9f450e142168a0c289c758be8c32ded009d671b\": container with ID starting with ebac456974f0d627615bde6ce9f450e142168a0c289c758be8c32ded009d671b not found: ID does not exist" Aug 01 00:48:44.130679 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:44.130652 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg"] Aug 01 00:48:44.135167 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:44.135143 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-852cg"] Aug 01 00:48:45.728599 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:48:45.728562 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a582a43b-f0c6-40d4-8550-80a7f1dba309" path="/var/lib/kubelet/pods/a582a43b-f0c6-40d4-8550-80a7f1dba309/volumes" Aug 01 00:49:43.773924 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.773885 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb"] Aug 01 00:49:43.774347 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.774311 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c787ccd8-b889-48ec-afe2-3c900e24c4e9" containerName="cleanup" Aug 01 00:49:43.774347 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.774323 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c787ccd8-b889-48ec-afe2-3c900e24c4e9" containerName="cleanup" Aug 01 00:49:43.774421 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.774357 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a582a43b-f0c6-40d4-8550-80a7f1dba309" containerName="manager" Aug 01 00:49:43.774421 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.774363 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a582a43b-f0c6-40d4-8550-80a7f1dba309" containerName="manager" Aug 01 00:49:43.774483 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.774422 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a582a43b-f0c6-40d4-8550-80a7f1dba309" containerName="manager" Aug 01 00:49:43.774483 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.774436 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c787ccd8-b889-48ec-afe2-3c900e24c4e9" containerName="cleanup" Aug 01 00:49:43.778624 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.778606 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" Aug 01 00:49:43.781509 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.781473 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 01 00:49:43.781509 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.781498 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 01 00:49:43.781696 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.781510 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dnhvv\"" Aug 01 00:49:43.789401 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.789372 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb"] Aug 01 00:49:43.842811 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.842765 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb6rf\" (UniqueName: \"kubernetes.io/projected/e17f5cdc-37d0-47b2-8caa-4b704458feb9-kube-api-access-nb6rf\") pod \"kuadrant-operator-controller-manager-68cf589589-qm8wb\" (UID: \"e17f5cdc-37d0-47b2-8caa-4b704458feb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" Aug 01 00:49:43.842990 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.842902 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e17f5cdc-37d0-47b2-8caa-4b704458feb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-qm8wb\" (UID: \"e17f5cdc-37d0-47b2-8caa-4b704458feb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" Aug 01 00:49:43.943848 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.943804 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6rf\" (UniqueName: \"kubernetes.io/projected/e17f5cdc-37d0-47b2-8caa-4b704458feb9-kube-api-access-nb6rf\") pod \"kuadrant-operator-controller-manager-68cf589589-qm8wb\" (UID: \"e17f5cdc-37d0-47b2-8caa-4b704458feb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" Aug 01 00:49:43.944049 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.943929 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e17f5cdc-37d0-47b2-8caa-4b704458feb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-qm8wb\" (UID: \"e17f5cdc-37d0-47b2-8caa-4b704458feb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" Aug 01 00:49:43.944347 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.944326 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e17f5cdc-37d0-47b2-8caa-4b704458feb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-qm8wb\" (UID: \"e17f5cdc-37d0-47b2-8caa-4b704458feb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" Aug 01 00:49:43.952577 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:43.952553 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6rf\" (UniqueName: \"kubernetes.io/projected/e17f5cdc-37d0-47b2-8caa-4b704458feb9-kube-api-access-nb6rf\") pod \"kuadrant-operator-controller-manager-68cf589589-qm8wb\" (UID: \"e17f5cdc-37d0-47b2-8caa-4b704458feb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" Aug 01 00:49:44.089297 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:44.089259 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" Aug 01 00:49:44.247619 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:44.247594 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb"] Aug 01 00:49:44.250158 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:49:44.250130 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17f5cdc_37d0_47b2_8caa_4b704458feb9.slice/crio-dd89afcbac4299a67200c9735df704108165a63a1d0c54a95ac8d252852a5f15 WatchSource:0}: Error finding container dd89afcbac4299a67200c9735df704108165a63a1d0c54a95ac8d252852a5f15: Status 404 returned error can't find the container with id dd89afcbac4299a67200c9735df704108165a63a1d0c54a95ac8d252852a5f15 Aug 01 00:49:44.393914 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:44.393824 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" event={"ID":"e17f5cdc-37d0-47b2-8caa-4b704458feb9","Type":"ContainerStarted","Data":"a6452df031e4b766ef60e6c1160664d08f2234d0f568946772f493abc797e955"} Aug 01 00:49:44.393914 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:44.393884 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" event={"ID":"e17f5cdc-37d0-47b2-8caa-4b704458feb9","Type":"ContainerStarted","Data":"dd89afcbac4299a67200c9735df704108165a63a1d0c54a95ac8d252852a5f15"} Aug 01 00:49:44.394136 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:44.393937 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" Aug 01 00:49:44.413345 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:44.413285 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" podStartSLOduration=1.4132685409999999 podStartE2EDuration="1.413268541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:49:44.412385692 +0000 UTC m=+1273.201107542" watchObservedRunningTime="2026-08-01 00:49:44.413268541 +0000 UTC m=+1273.201990386" Aug 01 00:49:55.401286 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:49:55.401252 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" Aug 01 00:53:31.750732 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:53:31.750702 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:53:31.753664 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:53:31.753633 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:53:31.756289 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:53:31.756268 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:53:31.759393 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:53:31.759375 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:58:12.973110 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:12.973073 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc"] Aug 01 00:58:12.976317 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:12.976297 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:12.978602 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:12.978580 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 01 00:58:12.979651 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:12.979634 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-202229e0-serving-cert\"" Aug 01 00:58:12.979710 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:12.979652 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 00:58:12.979710 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:12.979634 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gf2hx\"" Aug 01 00:58:12.984384 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:12.984362 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc"] Aug 01 00:58:13.087641 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.087608 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmbs\" (UniqueName: \"kubernetes.io/projected/547f10e5-5f5f-416d-be9f-f970d72c5e85-kube-api-access-tbmbs\") pod \"maas-api-e2e-ait-202229e0-76bb55464-9zhnc\" (UID: \"547f10e5-5f5f-416d-be9f-f970d72c5e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:13.087641 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.087644 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/547f10e5-5f5f-416d-be9f-f970d72c5e85-maas-api-tls\") pod \"maas-api-e2e-ait-202229e0-76bb55464-9zhnc\" (UID: \"547f10e5-5f5f-416d-be9f-f970d72c5e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:13.087875 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.087662 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/547f10e5-5f5f-416d-be9f-f970d72c5e85-openshift-service-ca\") pod \"maas-api-e2e-ait-202229e0-76bb55464-9zhnc\" (UID: \"547f10e5-5f5f-416d-be9f-f970d72c5e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:13.188742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.188707 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmbs\" (UniqueName: \"kubernetes.io/projected/547f10e5-5f5f-416d-be9f-f970d72c5e85-kube-api-access-tbmbs\") pod \"maas-api-e2e-ait-202229e0-76bb55464-9zhnc\" (UID: \"547f10e5-5f5f-416d-be9f-f970d72c5e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:13.188742 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.188748 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/547f10e5-5f5f-416d-be9f-f970d72c5e85-maas-api-tls\") pod \"maas-api-e2e-ait-202229e0-76bb55464-9zhnc\" (UID: \"547f10e5-5f5f-416d-be9f-f970d72c5e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:13.188988 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.188767 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/547f10e5-5f5f-416d-be9f-f970d72c5e85-openshift-service-ca\") pod \"maas-api-e2e-ait-202229e0-76bb55464-9zhnc\" (UID: \"547f10e5-5f5f-416d-be9f-f970d72c5e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:13.188988 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:58:13.188867 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-serving-cert: secret "maas-api-e2e-ait-202229e0-serving-cert" not found Aug 01 00:58:13.188988 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:58:13.188937 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/547f10e5-5f5f-416d-be9f-f970d72c5e85-maas-api-tls podName:547f10e5-5f5f-416d-be9f-f970d72c5e85 nodeName:}" failed. No retries permitted until 2026-08-01 00:58:13.688922205 +0000 UTC m=+1782.477644012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/547f10e5-5f5f-416d-be9f-f970d72c5e85-maas-api-tls") pod "maas-api-e2e-ait-202229e0-76bb55464-9zhnc" (UID: "547f10e5-5f5f-416d-be9f-f970d72c5e85") : secret "maas-api-e2e-ait-202229e0-serving-cert" not found Aug 01 00:58:13.189455 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.189437 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/547f10e5-5f5f-416d-be9f-f970d72c5e85-openshift-service-ca\") pod \"maas-api-e2e-ait-202229e0-76bb55464-9zhnc\" (UID: \"547f10e5-5f5f-416d-be9f-f970d72c5e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:13.197523 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.197498 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmbs\" (UniqueName: \"kubernetes.io/projected/547f10e5-5f5f-416d-be9f-f970d72c5e85-kube-api-access-tbmbs\") pod \"maas-api-e2e-ait-202229e0-76bb55464-9zhnc\" (UID: \"547f10e5-5f5f-416d-be9f-f970d72c5e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:13.634825 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.634782 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh"] Aug 01 00:58:13.638522 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.638506 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" Aug 01 00:58:13.643332 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.643308 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-202229e0-dockercfg-8drzv\"" Aug 01 00:58:13.643519 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.643499 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-202229e0\"" Aug 01 00:58:13.649831 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.649810 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh"] Aug 01 00:58:13.666384 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.666362 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r"] Aug 01 00:58:13.670023 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.670003 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" Aug 01 00:58:13.685210 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.685190 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r"] Aug 01 00:58:13.693510 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.693487 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxvnk\" (UniqueName: \"kubernetes.io/projected/51ac9152-6256-4b97-a052-cdd6ef2c26c3-kube-api-access-xxvnk\") pod \"payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh\" (UID: \"51ac9152-6256-4b97-a052-cdd6ef2c26c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" Aug 01 00:58:13.693621 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.693549 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cds4d\" (UniqueName: \"kubernetes.io/projected/47b5314f-ed19-4453-bf82-d297a745cf71-kube-api-access-cds4d\") pod \"payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r\" (UID: \"47b5314f-ed19-4453-bf82-d297a745cf71\") " pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" Aug 01 00:58:13.693621 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.693591 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47b5314f-ed19-4453-bf82-d297a745cf71-plugins-config-volume\") pod \"payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r\" (UID: \"47b5314f-ed19-4453-bf82-d297a745cf71\") " pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" Aug 01 00:58:13.693714 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.693617 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ac9152-6256-4b97-a052-cdd6ef2c26c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh\" (UID: \"51ac9152-6256-4b97-a052-cdd6ef2c26c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" Aug 01 00:58:13.693714 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.693669 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/547f10e5-5f5f-416d-be9f-f970d72c5e85-maas-api-tls\") pod \"maas-api-e2e-ait-202229e0-76bb55464-9zhnc\" (UID: \"547f10e5-5f5f-416d-be9f-f970d72c5e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:13.695975 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.695951 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/547f10e5-5f5f-416d-be9f-f970d72c5e85-maas-api-tls\") pod \"maas-api-e2e-ait-202229e0-76bb55464-9zhnc\" (UID: \"547f10e5-5f5f-416d-be9f-f970d72c5e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:13.794886 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.794844 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cds4d\" (UniqueName: \"kubernetes.io/projected/47b5314f-ed19-4453-bf82-d297a745cf71-kube-api-access-cds4d\") pod \"payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r\" (UID: \"47b5314f-ed19-4453-bf82-d297a745cf71\") " pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" Aug 01 00:58:13.794886 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.794893 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47b5314f-ed19-4453-bf82-d297a745cf71-plugins-config-volume\") pod \"payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r\" (UID: \"47b5314f-ed19-4453-bf82-d297a745cf71\") " pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" Aug 01 00:58:13.795138 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.794912 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ac9152-6256-4b97-a052-cdd6ef2c26c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh\" (UID: \"51ac9152-6256-4b97-a052-cdd6ef2c26c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" Aug 01 00:58:13.795138 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.794990 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxvnk\" (UniqueName: \"kubernetes.io/projected/51ac9152-6256-4b97-a052-cdd6ef2c26c3-kube-api-access-xxvnk\") pod \"payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh\" (UID: \"51ac9152-6256-4b97-a052-cdd6ef2c26c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" Aug 01 00:58:13.795719 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.795698 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ac9152-6256-4b97-a052-cdd6ef2c26c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh\" (UID: \"51ac9152-6256-4b97-a052-cdd6ef2c26c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" Aug 01 00:58:13.795719 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.795710 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47b5314f-ed19-4453-bf82-d297a745cf71-plugins-config-volume\") pod \"payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r\" (UID: \"47b5314f-ed19-4453-bf82-d297a745cf71\") " pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" Aug 01 00:58:13.806444 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.806401 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cds4d\" (UniqueName: \"kubernetes.io/projected/47b5314f-ed19-4453-bf82-d297a745cf71-kube-api-access-cds4d\") pod \"payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r\" (UID: \"47b5314f-ed19-4453-bf82-d297a745cf71\") " pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" Aug 01 00:58:13.806572 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.806404 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxvnk\" (UniqueName: \"kubernetes.io/projected/51ac9152-6256-4b97-a052-cdd6ef2c26c3-kube-api-access-xxvnk\") pod \"payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh\" (UID: \"51ac9152-6256-4b97-a052-cdd6ef2c26c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" Aug 01 00:58:13.888169 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.888040 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:13.949799 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.949766 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" Aug 01 00:58:13.981440 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:13.981400 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" Aug 01 00:58:14.056326 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:14.055782 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc"] Aug 01 00:58:14.062410 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:14.061239 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 00:58:14.124856 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:14.124832 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh"] Aug 01 00:58:14.128297 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:58:14.128259 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ac9152_6256_4b97_a052_cdd6ef2c26c3.slice/crio-54dde217681bcb6b126cd1701bf00c8e57fff45fd9ab021891fd22d3179580be WatchSource:0}: Error finding container 54dde217681bcb6b126cd1701bf00c8e57fff45fd9ab021891fd22d3179580be: Status 404 returned error can't find the container with id 54dde217681bcb6b126cd1701bf00c8e57fff45fd9ab021891fd22d3179580be Aug 01 00:58:14.153821 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:14.153792 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r"] Aug 01 00:58:14.162175 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:58:14.162151 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b5314f_ed19_4453_bf82_d297a745cf71.slice/crio-6df1dca896f360408c50c21cd5df60b6e593e7968785689e89b4d3d2fef7b564 WatchSource:0}: Error finding container 6df1dca896f360408c50c21cd5df60b6e593e7968785689e89b4d3d2fef7b564: Status 404 returned error can't find the container with id 6df1dca896f360408c50c21cd5df60b6e593e7968785689e89b4d3d2fef7b564 Aug 01 00:58:14.786126 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:14.786090 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" event={"ID":"547f10e5-5f5f-416d-be9f-f970d72c5e85","Type":"ContainerStarted","Data":"2c990d1e517dd0d1af0182beecf5407b7926ddc08d69b720e5704f5b1bb3b1ad"} Aug 01 00:58:14.788312 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:14.788270 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" event={"ID":"47b5314f-ed19-4453-bf82-d297a745cf71","Type":"ContainerStarted","Data":"34cfb7e327a765a9f9fa8b6095fc111494049defc4de871a2641b5aa024b14e0"} Aug 01 00:58:14.788312 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:14.788322 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" event={"ID":"47b5314f-ed19-4453-bf82-d297a745cf71","Type":"ContainerStarted","Data":"6df1dca896f360408c50c21cd5df60b6e593e7968785689e89b4d3d2fef7b564"} Aug 01 00:58:14.788529 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:14.788380 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" Aug 01 00:58:14.790346 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:14.790305 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" event={"ID":"51ac9152-6256-4b97-a052-cdd6ef2c26c3","Type":"ContainerStarted","Data":"32ec74fda08be228a1a6a984328f5cc7a72157132d331d2075fb3efbaad418cb"} Aug 01 00:58:14.790346 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:14.790347 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" event={"ID":"51ac9152-6256-4b97-a052-cdd6ef2c26c3","Type":"ContainerStarted","Data":"54dde217681bcb6b126cd1701bf00c8e57fff45fd9ab021891fd22d3179580be"} Aug 01 00:58:14.790496 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:14.790483 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" Aug 01 00:58:14.804976 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:14.804930 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" podStartSLOduration=1.804915799 podStartE2EDuration="1.804915799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:58:14.803257935 +0000 UTC m=+1783.591979772" watchObservedRunningTime="2026-08-01 00:58:14.804915799 +0000 UTC m=+1783.593637624" Aug 01 00:58:14.827246 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:14.827202 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" podStartSLOduration=1.827185954 podStartE2EDuration="1.827185954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:58:14.825281285 +0000 UTC m=+1783.614003112" watchObservedRunningTime="2026-08-01 00:58:14.827185954 +0000 UTC m=+1783.615907759" Aug 01 00:58:15.799097 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:15.799032 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" event={"ID":"547f10e5-5f5f-416d-be9f-f970d72c5e85","Type":"ContainerStarted","Data":"724aaf4e45cf3410e81a6189a0a19c3aa5552c80f87e0496ebef30ff75ff7c35"} Aug 01 00:58:15.799529 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:15.799457 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:15.817769 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:15.817399 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" podStartSLOduration=2.297308133 podStartE2EDuration="3.817379144s" totalImagesPullingTime="1.520071011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:58:15.816726397 +0000 UTC m=+1784.605448247" watchObservedRunningTime="2026-08-01 00:58:15.817379144 +0000 UTC m=+1784.606100970" Aug 01 00:58:22.814321 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:22.814291 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:25.801658 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:25.801582 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" Aug 01 00:58:25.801658 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:25.801650 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" Aug 01 00:58:31.780843 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:31.780813 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:58:31.784298 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:31.784272 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 00:58:31.786822 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:31.786798 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:58:31.790120 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:31.790099 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 00:58:43.512884 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.512842 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc"] Aug 01 00:58:43.513746 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.513684 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" podUID="547f10e5-5f5f-416d-be9f-f970d72c5e85" containerName="maas-api" containerID="cri-o://724aaf4e45cf3410e81a6189a0a19c3aa5552c80f87e0496ebef30ff75ff7c35" gracePeriod=30 Aug 01 00:58:43.575516 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.575482 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r"] Aug 01 00:58:43.575772 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.575746 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" podUID="47b5314f-ed19-4453-bf82-d297a745cf71" containerName="payload-processing" containerID="cri-o://34cfb7e327a765a9f9fa8b6095fc111494049defc4de871a2641b5aa024b14e0" gracePeriod=30 Aug 01 00:58:43.581228 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.581199 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh"] Aug 01 00:58:43.581796 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.581604 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" podUID="51ac9152-6256-4b97-a052-cdd6ef2c26c3" containerName="payload-pre-processing" containerID="cri-o://32ec74fda08be228a1a6a984328f5cc7a72157132d331d2075fb3efbaad418cb" gracePeriod=30 Aug 01 00:58:43.864141 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.864114 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:43.877853 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.877833 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" Aug 01 00:58:43.883974 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.883950 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" Aug 01 00:58:43.938285 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.938246 2563 generic.go:356] "Generic (PLEG): container finished" podID="547f10e5-5f5f-416d-be9f-f970d72c5e85" containerID="724aaf4e45cf3410e81a6189a0a19c3aa5552c80f87e0496ebef30ff75ff7c35" exitCode=0 Aug 01 00:58:43.938589 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.938311 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" Aug 01 00:58:43.938589 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.938342 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" event={"ID":"547f10e5-5f5f-416d-be9f-f970d72c5e85","Type":"ContainerDied","Data":"724aaf4e45cf3410e81a6189a0a19c3aa5552c80f87e0496ebef30ff75ff7c35"} Aug 01 00:58:43.938589 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.938414 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc" event={"ID":"547f10e5-5f5f-416d-be9f-f970d72c5e85","Type":"ContainerDied","Data":"2c990d1e517dd0d1af0182beecf5407b7926ddc08d69b720e5704f5b1bb3b1ad"} Aug 01 00:58:43.938589 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.938447 2563 scope.go:117] "RemoveContainer" containerID="724aaf4e45cf3410e81a6189a0a19c3aa5552c80f87e0496ebef30ff75ff7c35" Aug 01 00:58:43.940197 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.940156 2563 generic.go:356] "Generic (PLEG): container finished" podID="47b5314f-ed19-4453-bf82-d297a745cf71" containerID="34cfb7e327a765a9f9fa8b6095fc111494049defc4de871a2641b5aa024b14e0" exitCode=0 Aug 01 00:58:43.940317 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.940232 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" Aug 01 00:58:43.940317 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.940241 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" event={"ID":"47b5314f-ed19-4453-bf82-d297a745cf71","Type":"ContainerDied","Data":"34cfb7e327a765a9f9fa8b6095fc111494049defc4de871a2641b5aa024b14e0"} Aug 01 00:58:43.940317 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.940285 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r" event={"ID":"47b5314f-ed19-4453-bf82-d297a745cf71","Type":"ContainerDied","Data":"6df1dca896f360408c50c21cd5df60b6e593e7968785689e89b4d3d2fef7b564"} Aug 01 00:58:43.941749 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.941722 2563 generic.go:356] "Generic (PLEG): container finished" podID="51ac9152-6256-4b97-a052-cdd6ef2c26c3" containerID="32ec74fda08be228a1a6a984328f5cc7a72157132d331d2075fb3efbaad418cb" exitCode=0 Aug 01 00:58:43.941852 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.941749 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" event={"ID":"51ac9152-6256-4b97-a052-cdd6ef2c26c3","Type":"ContainerDied","Data":"32ec74fda08be228a1a6a984328f5cc7a72157132d331d2075fb3efbaad418cb"} Aug 01 00:58:43.941852 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.941769 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" Aug 01 00:58:43.941852 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.941790 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh" event={"ID":"51ac9152-6256-4b97-a052-cdd6ef2c26c3","Type":"ContainerDied","Data":"54dde217681bcb6b126cd1701bf00c8e57fff45fd9ab021891fd22d3179580be"} Aug 01 00:58:43.947916 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.947900 2563 scope.go:117] "RemoveContainer" containerID="724aaf4e45cf3410e81a6189a0a19c3aa5552c80f87e0496ebef30ff75ff7c35" Aug 01 00:58:43.948233 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:58:43.948196 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724aaf4e45cf3410e81a6189a0a19c3aa5552c80f87e0496ebef30ff75ff7c35\": container with ID starting with 724aaf4e45cf3410e81a6189a0a19c3aa5552c80f87e0496ebef30ff75ff7c35 not found: ID does not exist" containerID="724aaf4e45cf3410e81a6189a0a19c3aa5552c80f87e0496ebef30ff75ff7c35" Aug 01 00:58:43.948387 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.948239 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724aaf4e45cf3410e81a6189a0a19c3aa5552c80f87e0496ebef30ff75ff7c35"} err="failed to get container status \"724aaf4e45cf3410e81a6189a0a19c3aa5552c80f87e0496ebef30ff75ff7c35\": rpc error: code = NotFound desc = could not find container \"724aaf4e45cf3410e81a6189a0a19c3aa5552c80f87e0496ebef30ff75ff7c35\": container with ID starting with 724aaf4e45cf3410e81a6189a0a19c3aa5552c80f87e0496ebef30ff75ff7c35 not found: ID does not exist" Aug 01 00:58:43.948387 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.948261 2563 scope.go:117] "RemoveContainer" containerID="34cfb7e327a765a9f9fa8b6095fc111494049defc4de871a2641b5aa024b14e0" Aug 01 00:58:43.956712 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.956694 2563 scope.go:117] "RemoveContainer" containerID="34cfb7e327a765a9f9fa8b6095fc111494049defc4de871a2641b5aa024b14e0" Aug 01 00:58:43.956942 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:58:43.956923 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34cfb7e327a765a9f9fa8b6095fc111494049defc4de871a2641b5aa024b14e0\": container with ID starting with 34cfb7e327a765a9f9fa8b6095fc111494049defc4de871a2641b5aa024b14e0 not found: ID does not exist" containerID="34cfb7e327a765a9f9fa8b6095fc111494049defc4de871a2641b5aa024b14e0" Aug 01 00:58:43.956997 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.956950 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34cfb7e327a765a9f9fa8b6095fc111494049defc4de871a2641b5aa024b14e0"} err="failed to get container status \"34cfb7e327a765a9f9fa8b6095fc111494049defc4de871a2641b5aa024b14e0\": rpc error: code = NotFound desc = could not find container \"34cfb7e327a765a9f9fa8b6095fc111494049defc4de871a2641b5aa024b14e0\": container with ID starting with 34cfb7e327a765a9f9fa8b6095fc111494049defc4de871a2641b5aa024b14e0 not found: ID does not exist" Aug 01 00:58:43.956997 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.956969 2563 scope.go:117] "RemoveContainer" containerID="32ec74fda08be228a1a6a984328f5cc7a72157132d331d2075fb3efbaad418cb" Aug 01 00:58:43.961176 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.961157 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxvnk\" (UniqueName: \"kubernetes.io/projected/51ac9152-6256-4b97-a052-cdd6ef2c26c3-kube-api-access-xxvnk\") pod \"51ac9152-6256-4b97-a052-cdd6ef2c26c3\" (UID: \"51ac9152-6256-4b97-a052-cdd6ef2c26c3\") " Aug 01 00:58:43.961259 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.961228 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cds4d\" (UniqueName: \"kubernetes.io/projected/47b5314f-ed19-4453-bf82-d297a745cf71-kube-api-access-cds4d\") pod \"47b5314f-ed19-4453-bf82-d297a745cf71\" (UID: \"47b5314f-ed19-4453-bf82-d297a745cf71\") " Aug 01 00:58:43.961259 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.961249 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47b5314f-ed19-4453-bf82-d297a745cf71-plugins-config-volume\") pod \"47b5314f-ed19-4453-bf82-d297a745cf71\" (UID: \"47b5314f-ed19-4453-bf82-d297a745cf71\") " Aug 01 00:58:43.961347 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.961271 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ac9152-6256-4b97-a052-cdd6ef2c26c3-plugins-config-volume\") pod \"51ac9152-6256-4b97-a052-cdd6ef2c26c3\" (UID: \"51ac9152-6256-4b97-a052-cdd6ef2c26c3\") " Aug 01 00:58:43.961347 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.961303 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbmbs\" (UniqueName: \"kubernetes.io/projected/547f10e5-5f5f-416d-be9f-f970d72c5e85-kube-api-access-tbmbs\") pod \"547f10e5-5f5f-416d-be9f-f970d72c5e85\" (UID: \"547f10e5-5f5f-416d-be9f-f970d72c5e85\") " Aug 01 00:58:43.961347 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.961339 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/547f10e5-5f5f-416d-be9f-f970d72c5e85-maas-api-tls\") pod \"547f10e5-5f5f-416d-be9f-f970d72c5e85\" (UID: \"547f10e5-5f5f-416d-be9f-f970d72c5e85\") " Aug 01 00:58:43.961496 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.961381 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/547f10e5-5f5f-416d-be9f-f970d72c5e85-openshift-service-ca\") pod \"547f10e5-5f5f-416d-be9f-f970d72c5e85\" (UID: \"547f10e5-5f5f-416d-be9f-f970d72c5e85\") " Aug 01 00:58:43.961941 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.961694 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ac9152-6256-4b97-a052-cdd6ef2c26c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51ac9152-6256-4b97-a052-cdd6ef2c26c3" (UID: "51ac9152-6256-4b97-a052-cdd6ef2c26c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:58:43.961941 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.961884 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547f10e5-5f5f-416d-be9f-f970d72c5e85-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "547f10e5-5f5f-416d-be9f-f970d72c5e85" (UID: "547f10e5-5f5f-416d-be9f-f970d72c5e85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:58:43.962172 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.961967 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b5314f-ed19-4453-bf82-d297a745cf71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "47b5314f-ed19-4453-bf82-d297a745cf71" (UID: "47b5314f-ed19-4453-bf82-d297a745cf71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:58:43.963552 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.963521 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547f10e5-5f5f-416d-be9f-f970d72c5e85-kube-api-access-tbmbs" (OuterVolumeSpecName: "kube-api-access-tbmbs") pod "547f10e5-5f5f-416d-be9f-f970d72c5e85" (UID: "547f10e5-5f5f-416d-be9f-f970d72c5e85"). InnerVolumeSpecName "kube-api-access-tbmbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:58:43.963851 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.963828 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ac9152-6256-4b97-a052-cdd6ef2c26c3-kube-api-access-xxvnk" (OuterVolumeSpecName: "kube-api-access-xxvnk") pod "51ac9152-6256-4b97-a052-cdd6ef2c26c3" (UID: "51ac9152-6256-4b97-a052-cdd6ef2c26c3"). InnerVolumeSpecName "kube-api-access-xxvnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:58:43.963921 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.963855 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b5314f-ed19-4453-bf82-d297a745cf71-kube-api-access-cds4d" (OuterVolumeSpecName: "kube-api-access-cds4d") pod "47b5314f-ed19-4453-bf82-d297a745cf71" (UID: "47b5314f-ed19-4453-bf82-d297a745cf71"). InnerVolumeSpecName "kube-api-access-cds4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:58:43.963921 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.963859 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547f10e5-5f5f-416d-be9f-f970d72c5e85-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "547f10e5-5f5f-416d-be9f-f970d72c5e85" (UID: "547f10e5-5f5f-416d-be9f-f970d72c5e85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:58:43.965212 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.965197 2563 scope.go:117] "RemoveContainer" containerID="32ec74fda08be228a1a6a984328f5cc7a72157132d331d2075fb3efbaad418cb" Aug 01 00:58:43.965422 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:58:43.965405 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ec74fda08be228a1a6a984328f5cc7a72157132d331d2075fb3efbaad418cb\": container with ID starting with 32ec74fda08be228a1a6a984328f5cc7a72157132d331d2075fb3efbaad418cb not found: ID does not exist" containerID="32ec74fda08be228a1a6a984328f5cc7a72157132d331d2075fb3efbaad418cb" Aug 01 00:58:43.965466 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:43.965433 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ec74fda08be228a1a6a984328f5cc7a72157132d331d2075fb3efbaad418cb"} err="failed to get container status \"32ec74fda08be228a1a6a984328f5cc7a72157132d331d2075fb3efbaad418cb\": rpc error: code = NotFound desc = could not find container \"32ec74fda08be228a1a6a984328f5cc7a72157132d331d2075fb3efbaad418cb\": container with ID starting with 32ec74fda08be228a1a6a984328f5cc7a72157132d331d2075fb3efbaad418cb not found: ID does not exist" Aug 01 00:58:44.062396 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.062367 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/547f10e5-5f5f-416d-be9f-f970d72c5e85-maas-api-tls\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:58:44.062396 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.062396 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/547f10e5-5f5f-416d-be9f-f970d72c5e85-openshift-service-ca\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:58:44.062584 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.062407 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxvnk\" (UniqueName: \"kubernetes.io/projected/51ac9152-6256-4b97-a052-cdd6ef2c26c3-kube-api-access-xxvnk\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:58:44.062584 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.062418 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cds4d\" (UniqueName: \"kubernetes.io/projected/47b5314f-ed19-4453-bf82-d297a745cf71-kube-api-access-cds4d\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:58:44.062584 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.062427 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47b5314f-ed19-4453-bf82-d297a745cf71-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:58:44.062584 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.062435 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ac9152-6256-4b97-a052-cdd6ef2c26c3-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:58:44.062584 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.062443 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbmbs\" (UniqueName: \"kubernetes.io/projected/547f10e5-5f5f-416d-be9f-f970d72c5e85-kube-api-access-tbmbs\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:58:44.272225 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.272191 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc"] Aug 01 00:58:44.274444 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.274422 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-202229e0-76bb55464-9zhnc"] Aug 01 00:58:44.284899 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.284877 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r"] Aug 01 00:58:44.296518 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.296490 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-202229e0-7759459bb8-mkf5r"] Aug 01 00:58:44.307244 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.307217 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh"] Aug 01 00:58:44.309249 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.309226 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-202229e0-6c7dbd4fb-ktbnh"] Aug 01 00:58:44.792089 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.791999 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt"] Aug 01 00:58:44.792430 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.792360 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ac9152-6256-4b97-a052-cdd6ef2c26c3" containerName="payload-pre-processing" Aug 01 00:58:44.792430 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.792373 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ac9152-6256-4b97-a052-cdd6ef2c26c3" containerName="payload-pre-processing" Aug 01 00:58:44.792430 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.792390 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b5314f-ed19-4453-bf82-d297a745cf71" containerName="payload-processing" Aug 01 00:58:44.792430 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.792396 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b5314f-ed19-4453-bf82-d297a745cf71" containerName="payload-processing" Aug 01 00:58:44.792430 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.792410 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="547f10e5-5f5f-416d-be9f-f970d72c5e85" containerName="maas-api" Aug 01 00:58:44.792430 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.792416 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="547f10e5-5f5f-416d-be9f-f970d72c5e85" containerName="maas-api" Aug 01 00:58:44.792616 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.792477 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="51ac9152-6256-4b97-a052-cdd6ef2c26c3" containerName="payload-pre-processing" Aug 01 00:58:44.792616 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.792486 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b5314f-ed19-4453-bf82-d297a745cf71" containerName="payload-processing" Aug 01 00:58:44.792616 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.792493 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="547f10e5-5f5f-416d-be9f-f970d72c5e85" containerName="maas-api" Aug 01 00:58:44.796959 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.796938 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.799406 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.799387 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d5eeee24-gw-openshift-default-dockercfg-9nxng\"" Aug 01 00:58:44.806744 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.806720 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt"] Aug 01 00:58:44.870372 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.870335 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-istio-data\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.870521 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.870402 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-workload-certs\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.870521 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.870432 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-istio-envoy\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.870521 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.870455 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad9be01-7425-485d-a2fd-312de890129c-istio-podinfo\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.870627 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.870531 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad9be01-7425-485d-a2fd-312de890129c-istio-token\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.870671 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.870646 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad9be01-7425-485d-a2fd-312de890129c-istiod-ca-cert\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.870768 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.870755 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfq55\" (UniqueName: \"kubernetes.io/projected/3ad9be01-7425-485d-a2fd-312de890129c-kube-api-access-gfq55\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.870814 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.870782 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-workload-socket\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.870861 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.870846 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-credential-socket\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.971646 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.971607 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad9be01-7425-485d-a2fd-312de890129c-istiod-ca-cert\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.971862 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.971653 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfq55\" (UniqueName: \"kubernetes.io/projected/3ad9be01-7425-485d-a2fd-312de890129c-kube-api-access-gfq55\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.971862 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.971686 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-workload-socket\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.971862 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.971715 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-credential-socket\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.971862 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.971774 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-istio-data\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.971862 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.971820 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-workload-certs\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.972181 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.971940 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-istio-envoy\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.972181 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.971970 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad9be01-7425-485d-a2fd-312de890129c-istio-podinfo\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.972181 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.972004 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad9be01-7425-485d-a2fd-312de890129c-istio-token\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.972181 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.972136 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-workload-socket\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.972427 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.972201 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-credential-socket\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.972427 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.972235 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-istio-data\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.972427 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.972284 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-workload-certs\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.972427 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.972380 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad9be01-7425-485d-a2fd-312de890129c-istiod-ca-cert\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.974242 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.974221 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-istio-envoy\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.974460 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.974444 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad9be01-7425-485d-a2fd-312de890129c-istio-podinfo\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.982308 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.982288 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad9be01-7425-485d-a2fd-312de890129c-istio-token\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:44.982605 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:44.982583 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfq55\" (UniqueName: \"kubernetes.io/projected/3ad9be01-7425-485d-a2fd-312de890129c-kube-api-access-gfq55\") pod \"e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:45.109181 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:45.109148 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:45.248990 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:45.248964 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt"] Aug 01 00:58:45.250366 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:58:45.250340 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad9be01_7425_485d_a2fd_312de890129c.slice/crio-f229ce746d1ce0ded70435d3b0cf0de43d1f7b5d1df441e63eba662148946409 WatchSource:0}: Error finding container f229ce746d1ce0ded70435d3b0cf0de43d1f7b5d1df441e63eba662148946409: Status 404 returned error can't find the container with id f229ce746d1ce0ded70435d3b0cf0de43d1f7b5d1df441e63eba662148946409 Aug 01 00:58:45.727814 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:45.727783 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47b5314f-ed19-4453-bf82-d297a745cf71" path="/var/lib/kubelet/pods/47b5314f-ed19-4453-bf82-d297a745cf71/volumes" Aug 01 00:58:45.728196 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:45.728180 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51ac9152-6256-4b97-a052-cdd6ef2c26c3" path="/var/lib/kubelet/pods/51ac9152-6256-4b97-a052-cdd6ef2c26c3/volumes" Aug 01 00:58:45.728542 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:45.728530 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="547f10e5-5f5f-416d-be9f-f970d72c5e85" path="/var/lib/kubelet/pods/547f10e5-5f5f-416d-be9f-f970d72c5e85/volumes" Aug 01 00:58:45.955901 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:45.955786 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" event={"ID":"3ad9be01-7425-485d-a2fd-312de890129c","Type":"ContainerStarted","Data":"f229ce746d1ce0ded70435d3b0cf0de43d1f7b5d1df441e63eba662148946409"} Aug 01 00:58:48.313001 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:48.312956 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:58:48.313292 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:48.313048 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:58:48.313292 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:48.313114 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:58:48.972975 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:48.972931 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" event={"ID":"3ad9be01-7425-485d-a2fd-312de890129c","Type":"ContainerStarted","Data":"f638bf7e86b996c2a0a136f32fe79138a52edc9c65fb49f89ee9cca1a38b3ec1"} Aug 01 00:58:48.996632 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:48.996584 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" podStartSLOduration=1.936280835 podStartE2EDuration="4.996567728s" totalImagesPullingTime="3.060286893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:58:48.995536269 +0000 UTC m=+1817.784258097" watchObservedRunningTime="2026-08-01 00:58:48.996567728 +0000 UTC m=+1817.785289557" Aug 01 00:58:49.109929 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:49.109892 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:49.115369 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:49.115341 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:49.977565 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:49.977537 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:49.978500 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:49.978477 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:58:50.516611 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.516579 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498"] Aug 01 00:58:50.520796 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.520771 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:58:50.523437 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.523410 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 00:58:50.523562 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.523439 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 01 00:58:50.523562 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.523481 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d5eeee24-serving-cert\"" Aug 01 00:58:50.524408 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.524388 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gf2hx\"" Aug 01 00:58:50.528731 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.528708 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498"] Aug 01 00:58:50.628827 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.628797 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbmvt\" (UniqueName: \"kubernetes.io/projected/6527228d-44f3-48a2-8153-78da9b4ce446-kube-api-access-jbmvt\") pod \"maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498\" (UID: \"6527228d-44f3-48a2-8153-78da9b4ce446\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:58:50.628827 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.628840 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6527228d-44f3-48a2-8153-78da9b4ce446-maas-api-tls\") pod \"maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498\" (UID: \"6527228d-44f3-48a2-8153-78da9b4ce446\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:58:50.629032 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.628925 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6527228d-44f3-48a2-8153-78da9b4ce446-openshift-service-ca\") pod \"maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498\" (UID: \"6527228d-44f3-48a2-8153-78da9b4ce446\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:58:50.729652 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.729616 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbmvt\" (UniqueName: \"kubernetes.io/projected/6527228d-44f3-48a2-8153-78da9b4ce446-kube-api-access-jbmvt\") pod \"maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498\" (UID: \"6527228d-44f3-48a2-8153-78da9b4ce446\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:58:50.729808 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.729661 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6527228d-44f3-48a2-8153-78da9b4ce446-maas-api-tls\") pod \"maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498\" (UID: \"6527228d-44f3-48a2-8153-78da9b4ce446\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:58:50.729808 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.729709 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6527228d-44f3-48a2-8153-78da9b4ce446-openshift-service-ca\") pod \"maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498\" (UID: \"6527228d-44f3-48a2-8153-78da9b4ce446\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:58:50.729878 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:58:50.729829 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-serving-cert: secret "maas-api-e2e-migrate-d5eeee24-serving-cert" not found Aug 01 00:58:50.729965 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:58:50.729888 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6527228d-44f3-48a2-8153-78da9b4ce446-maas-api-tls podName:6527228d-44f3-48a2-8153-78da9b4ce446 nodeName:}" failed. No retries permitted until 2026-08-01 00:58:51.229874361 +0000 UTC m=+1820.018596165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6527228d-44f3-48a2-8153-78da9b4ce446-maas-api-tls") pod "maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" (UID: "6527228d-44f3-48a2-8153-78da9b4ce446") : secret "maas-api-e2e-migrate-d5eeee24-serving-cert" not found Aug 01 00:58:50.730348 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.730331 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6527228d-44f3-48a2-8153-78da9b4ce446-openshift-service-ca\") pod \"maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498\" (UID: \"6527228d-44f3-48a2-8153-78da9b4ce446\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:58:50.738014 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:50.737989 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbmvt\" (UniqueName: \"kubernetes.io/projected/6527228d-44f3-48a2-8153-78da9b4ce446-kube-api-access-jbmvt\") pod \"maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498\" (UID: \"6527228d-44f3-48a2-8153-78da9b4ce446\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:58:51.236161 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.236117 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6527228d-44f3-48a2-8153-78da9b4ce446-maas-api-tls\") pod \"maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498\" (UID: \"6527228d-44f3-48a2-8153-78da9b4ce446\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:58:51.238624 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.238597 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6527228d-44f3-48a2-8153-78da9b4ce446-maas-api-tls\") pod \"maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498\" (UID: \"6527228d-44f3-48a2-8153-78da9b4ce446\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:58:51.280223 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.280187 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45"] Aug 01 00:58:51.284019 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.284001 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" Aug 01 00:58:51.287067 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.287030 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d5eeee24-dockercfg-jwmp9\"" Aug 01 00:58:51.287368 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.287348 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d5eeee24\"" Aug 01 00:58:51.295385 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.295364 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45"] Aug 01 00:58:51.435689 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.435646 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:58:51.438034 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.438007 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2c28\" (UniqueName: \"kubernetes.io/projected/3f01e4d0-c115-4ea6-8551-dda951949990-kube-api-access-f2c28\") pod \"payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45\" (UID: \"3f01e4d0-c115-4ea6-8551-dda951949990\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" Aug 01 00:58:51.438179 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.438078 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f01e4d0-c115-4ea6-8551-dda951949990-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45\" (UID: \"3f01e4d0-c115-4ea6-8551-dda951949990\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" Aug 01 00:58:51.538978 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.538934 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f01e4d0-c115-4ea6-8551-dda951949990-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45\" (UID: \"3f01e4d0-c115-4ea6-8551-dda951949990\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" Aug 01 00:58:51.539173 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.539014 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2c28\" (UniqueName: \"kubernetes.io/projected/3f01e4d0-c115-4ea6-8551-dda951949990-kube-api-access-f2c28\") pod \"payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45\" (UID: \"3f01e4d0-c115-4ea6-8551-dda951949990\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" Aug 01 00:58:51.539831 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.539810 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f01e4d0-c115-4ea6-8551-dda951949990-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45\" (UID: \"3f01e4d0-c115-4ea6-8551-dda951949990\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" Aug 01 00:58:51.547327 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.547291 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2c28\" (UniqueName: \"kubernetes.io/projected/3f01e4d0-c115-4ea6-8551-dda951949990-kube-api-access-f2c28\") pod \"payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45\" (UID: \"3f01e4d0-c115-4ea6-8551-dda951949990\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" Aug 01 00:58:51.572613 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.572587 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498"] Aug 01 00:58:51.574730 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:58:51.574703 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6527228d_44f3_48a2_8153_78da9b4ce446.slice/crio-6972f9eba3fabe3bdf55afcffe9425b27a3455fd7db41f1e6c463e65466b24b9 WatchSource:0}: Error finding container 6972f9eba3fabe3bdf55afcffe9425b27a3455fd7db41f1e6c463e65466b24b9: Status 404 returned error can't find the container with id 6972f9eba3fabe3bdf55afcffe9425b27a3455fd7db41f1e6c463e65466b24b9 Aug 01 00:58:51.596307 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.596283 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" Aug 01 00:58:51.730115 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.729982 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45"] Aug 01 00:58:51.732303 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:58:51.732277 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f01e4d0_c115_4ea6_8551_dda951949990.slice/crio-4a8fcb5183c476004978d4ab33cc4ffd895f205efd787358a2236c17912c24a4 WatchSource:0}: Error finding container 4a8fcb5183c476004978d4ab33cc4ffd895f205efd787358a2236c17912c24a4: Status 404 returned error can't find the container with id 4a8fcb5183c476004978d4ab33cc4ffd895f205efd787358a2236c17912c24a4 Aug 01 00:58:51.989028 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.988994 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" event={"ID":"3f01e4d0-c115-4ea6-8551-dda951949990","Type":"ContainerStarted","Data":"1069ebf1855739d1a1f6c7ce2a5d28f1ff7cff857c56857f144232a223ffafa6"} Aug 01 00:58:51.989200 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.989083 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" Aug 01 00:58:51.989200 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.989112 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" event={"ID":"3f01e4d0-c115-4ea6-8551-dda951949990","Type":"ContainerStarted","Data":"4a8fcb5183c476004978d4ab33cc4ffd895f205efd787358a2236c17912c24a4"} Aug 01 00:58:51.990267 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:51.990238 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" event={"ID":"6527228d-44f3-48a2-8153-78da9b4ce446","Type":"ContainerStarted","Data":"6972f9eba3fabe3bdf55afcffe9425b27a3455fd7db41f1e6c463e65466b24b9"} Aug 01 00:58:52.996223 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:52.996187 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" event={"ID":"6527228d-44f3-48a2-8153-78da9b4ce446","Type":"ContainerStarted","Data":"f40146ebb63694919648dfeb2cf2182464b126aa8775a81d50dd4d4450b8fcc7"} Aug 01 00:58:53.016772 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:53.016711 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" podStartSLOduration=2.016692181 podStartE2EDuration="2.016692181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:58:52.005631525 +0000 UTC m=+1820.794353352" watchObservedRunningTime="2026-08-01 00:58:53.016692181 +0000 UTC m=+1821.805414010" Aug 01 00:58:53.017093 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:53.017051 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" podStartSLOduration=2.640519098 podStartE2EDuration="3.017044315s" totalImagesPullingTime="376.525217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:58:53.014049691 +0000 UTC m=+1821.802771518" watchObservedRunningTime="2026-08-01 00:58:53.017044315 +0000 UTC m=+1821.805766147" Aug 01 00:58:54.001538 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:58:54.001509 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:59:00.011279 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:00.011245 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:59:02.998365 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:02.998335 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" Aug 01 00:59:04.065259 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:04.065231 2563 generic.go:356] "Generic (PLEG): container finished" podID="3ad9be01-7425-485d-a2fd-312de890129c" containerID="f638bf7e86b996c2a0a136f32fe79138a52edc9c65fb49f89ee9cca1a38b3ec1" exitCode=0 Aug 01 00:59:04.065602 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:04.065304 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" event={"ID":"3ad9be01-7425-485d-a2fd-312de890129c","Type":"ContainerDied","Data":"f638bf7e86b996c2a0a136f32fe79138a52edc9c65fb49f89ee9cca1a38b3ec1"} Aug 01 00:59:04.065695 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:04.065680 2563 scope.go:117] "RemoveContainer" containerID="f638bf7e86b996c2a0a136f32fe79138a52edc9c65fb49f89ee9cca1a38b3ec1" Aug 01 00:59:04.066619 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:04.066593 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:59:04.066678 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:04.066666 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:59:04.066716 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:04.066706 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:59:04.977943 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:04.977910 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:59:05.072773 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:05.072731 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" event={"ID":"3ad9be01-7425-485d-a2fd-312de890129c","Type":"ContainerStarted","Data":"b80ff522770823dabf4791931f73ea698ccb008477511c742b8054f14cdf9124"} Aug 01 00:59:05.109484 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:05.109461 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:59:05.110474 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:05.110445 2563 patch_prober.go:28] interesting pod/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Aug 01 00:59:05.110568 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:05.110486 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" podUID="3ad9be01-7425-485d-a2fd-312de890129c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Aug 01 00:59:06.110200 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:06.110166 2563 patch_prober.go:28] interesting pod/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Aug 01 00:59:06.110558 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:06.110234 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" podUID="3ad9be01-7425-485d-a2fd-312de890129c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Aug 01 00:59:07.110550 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:07.110513 2563 patch_prober.go:28] interesting pod/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Aug 01 00:59:07.110940 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:07.110589 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" podUID="3ad9be01-7425-485d-a2fd-312de890129c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Aug 01 00:59:08.109937 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:08.109905 2563 patch_prober.go:28] interesting pod/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Aug 01 00:59:08.110138 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:08.109957 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" podUID="3ad9be01-7425-485d-a2fd-312de890129c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Aug 01 00:59:09.109793 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:09.109762 2563 patch_prober.go:28] interesting pod/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Aug 01 00:59:09.110185 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:09.109819 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" podUID="3ad9be01-7425-485d-a2fd-312de890129c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Aug 01 00:59:10.114948 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:10.114918 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:59:10.115418 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:10.115329 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:59:10.116010 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:10.115980 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:59:21.052729 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.052688 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498"] Aug 01 00:59:21.053582 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.053550 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" podUID="6527228d-44f3-48a2-8153-78da9b4ce446" containerName="maas-api" containerID="cri-o://f40146ebb63694919648dfeb2cf2182464b126aa8775a81d50dd4d4450b8fcc7" gracePeriod=30 Aug 01 00:59:21.109747 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.109716 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45"] Aug 01 00:59:21.109963 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.109936 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" podUID="3f01e4d0-c115-4ea6-8551-dda951949990" containerName="payload-pre-processing" containerID="cri-o://1069ebf1855739d1a1f6c7ce2a5d28f1ff7cff857c56857f144232a223ffafa6" gracePeriod=30 Aug 01 00:59:21.359661 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.359630 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt"] Aug 01 00:59:21.360260 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.359871 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" podUID="3ad9be01-7425-485d-a2fd-312de890129c" containerName="istio-proxy" containerID="cri-o://b80ff522770823dabf4791931f73ea698ccb008477511c742b8054f14cdf9124" gracePeriod=30 Aug 01 00:59:21.368357 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.368333 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:59:21.412347 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.411999 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" Aug 01 00:59:21.422683 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.422606 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbmvt\" (UniqueName: \"kubernetes.io/projected/6527228d-44f3-48a2-8153-78da9b4ce446-kube-api-access-jbmvt\") pod \"6527228d-44f3-48a2-8153-78da9b4ce446\" (UID: \"6527228d-44f3-48a2-8153-78da9b4ce446\") " Aug 01 00:59:21.422885 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.422747 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6527228d-44f3-48a2-8153-78da9b4ce446-openshift-service-ca\") pod \"6527228d-44f3-48a2-8153-78da9b4ce446\" (UID: \"6527228d-44f3-48a2-8153-78da9b4ce446\") " Aug 01 00:59:21.422885 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.422813 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6527228d-44f3-48a2-8153-78da9b4ce446-maas-api-tls\") pod \"6527228d-44f3-48a2-8153-78da9b4ce446\" (UID: \"6527228d-44f3-48a2-8153-78da9b4ce446\") " Aug 01 00:59:21.423442 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.423408 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6527228d-44f3-48a2-8153-78da9b4ce446-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6527228d-44f3-48a2-8153-78da9b4ce446" (UID: "6527228d-44f3-48a2-8153-78da9b4ce446"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:59:21.425504 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.425470 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6527228d-44f3-48a2-8153-78da9b4ce446-kube-api-access-jbmvt" (OuterVolumeSpecName: "kube-api-access-jbmvt") pod "6527228d-44f3-48a2-8153-78da9b4ce446" (UID: "6527228d-44f3-48a2-8153-78da9b4ce446"). InnerVolumeSpecName "kube-api-access-jbmvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:59:21.429373 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.427008 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6527228d-44f3-48a2-8153-78da9b4ce446-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6527228d-44f3-48a2-8153-78da9b4ce446" (UID: "6527228d-44f3-48a2-8153-78da9b4ce446"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:59:21.524066 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.524030 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f01e4d0-c115-4ea6-8551-dda951949990-plugins-config-volume\") pod \"3f01e4d0-c115-4ea6-8551-dda951949990\" (UID: \"3f01e4d0-c115-4ea6-8551-dda951949990\") " Aug 01 00:59:21.524245 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.524169 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2c28\" (UniqueName: \"kubernetes.io/projected/3f01e4d0-c115-4ea6-8551-dda951949990-kube-api-access-f2c28\") pod \"3f01e4d0-c115-4ea6-8551-dda951949990\" (UID: \"3f01e4d0-c115-4ea6-8551-dda951949990\") " Aug 01 00:59:21.524403 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.524382 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f01e4d0-c115-4ea6-8551-dda951949990-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f01e4d0-c115-4ea6-8551-dda951949990" (UID: "3f01e4d0-c115-4ea6-8551-dda951949990"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:59:21.524446 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.524434 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6527228d-44f3-48a2-8153-78da9b4ce446-maas-api-tls\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:21.524480 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.524449 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbmvt\" (UniqueName: \"kubernetes.io/projected/6527228d-44f3-48a2-8153-78da9b4ce446-kube-api-access-jbmvt\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:21.524480 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.524459 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6527228d-44f3-48a2-8153-78da9b4ce446-openshift-service-ca\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:21.526136 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.526116 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f01e4d0-c115-4ea6-8551-dda951949990-kube-api-access-f2c28" (OuterVolumeSpecName: "kube-api-access-f2c28") pod "3f01e4d0-c115-4ea6-8551-dda951949990" (UID: "3f01e4d0-c115-4ea6-8551-dda951949990"). InnerVolumeSpecName "kube-api-access-f2c28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:59:21.624946 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.624866 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f01e4d0-c115-4ea6-8551-dda951949990-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:21.624946 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:21.624894 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2c28\" (UniqueName: \"kubernetes.io/projected/3f01e4d0-c115-4ea6-8551-dda951949990-kube-api-access-f2c28\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:22.113745 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.113711 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp"] Aug 01 00:59:22.114115 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.114102 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f01e4d0-c115-4ea6-8551-dda951949990" containerName="payload-pre-processing" Aug 01 00:59:22.114115 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.114117 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f01e4d0-c115-4ea6-8551-dda951949990" containerName="payload-pre-processing" Aug 01 00:59:22.114193 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.114128 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6527228d-44f3-48a2-8153-78da9b4ce446" containerName="maas-api" Aug 01 00:59:22.114193 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.114134 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6527228d-44f3-48a2-8153-78da9b4ce446" containerName="maas-api" Aug 01 00:59:22.114252 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.114192 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6527228d-44f3-48a2-8153-78da9b4ce446" containerName="maas-api" Aug 01 00:59:22.114252 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.114203 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f01e4d0-c115-4ea6-8551-dda951949990" containerName="payload-pre-processing" Aug 01 00:59:22.117510 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.117486 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.119917 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.119897 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-99a0aaf4-openshift-default-dockercfg-2nb9v\"" Aug 01 00:59:22.131173 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.131144 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp"] Aug 01 00:59:22.167533 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.167499 2563 generic.go:356] "Generic (PLEG): container finished" podID="3f01e4d0-c115-4ea6-8551-dda951949990" containerID="1069ebf1855739d1a1f6c7ce2a5d28f1ff7cff857c56857f144232a223ffafa6" exitCode=0 Aug 01 00:59:22.167698 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.167567 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" Aug 01 00:59:22.167698 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.167604 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" event={"ID":"3f01e4d0-c115-4ea6-8551-dda951949990","Type":"ContainerDied","Data":"1069ebf1855739d1a1f6c7ce2a5d28f1ff7cff857c56857f144232a223ffafa6"} Aug 01 00:59:22.167698 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.167673 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45" event={"ID":"3f01e4d0-c115-4ea6-8551-dda951949990","Type":"ContainerDied","Data":"4a8fcb5183c476004978d4ab33cc4ffd895f205efd787358a2236c17912c24a4"} Aug 01 00:59:22.167842 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.167710 2563 scope.go:117] "RemoveContainer" containerID="1069ebf1855739d1a1f6c7ce2a5d28f1ff7cff857c56857f144232a223ffafa6" Aug 01 00:59:22.169099 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.169074 2563 generic.go:356] "Generic (PLEG): container finished" podID="6527228d-44f3-48a2-8153-78da9b4ce446" containerID="f40146ebb63694919648dfeb2cf2182464b126aa8775a81d50dd4d4450b8fcc7" exitCode=0 Aug 01 00:59:22.169216 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.169117 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" event={"ID":"6527228d-44f3-48a2-8153-78da9b4ce446","Type":"ContainerDied","Data":"f40146ebb63694919648dfeb2cf2182464b126aa8775a81d50dd4d4450b8fcc7"} Aug 01 00:59:22.169216 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.169156 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" event={"ID":"6527228d-44f3-48a2-8153-78da9b4ce446","Type":"ContainerDied","Data":"6972f9eba3fabe3bdf55afcffe9425b27a3455fd7db41f1e6c463e65466b24b9"} Aug 01 00:59:22.169216 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.169176 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498" Aug 01 00:59:22.177504 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.177486 2563 scope.go:117] "RemoveContainer" containerID="1069ebf1855739d1a1f6c7ce2a5d28f1ff7cff857c56857f144232a223ffafa6" Aug 01 00:59:22.177744 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:59:22.177726 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1069ebf1855739d1a1f6c7ce2a5d28f1ff7cff857c56857f144232a223ffafa6\": container with ID starting with 1069ebf1855739d1a1f6c7ce2a5d28f1ff7cff857c56857f144232a223ffafa6 not found: ID does not exist" containerID="1069ebf1855739d1a1f6c7ce2a5d28f1ff7cff857c56857f144232a223ffafa6" Aug 01 00:59:22.177795 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.177753 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1069ebf1855739d1a1f6c7ce2a5d28f1ff7cff857c56857f144232a223ffafa6"} err="failed to get container status \"1069ebf1855739d1a1f6c7ce2a5d28f1ff7cff857c56857f144232a223ffafa6\": rpc error: code = NotFound desc = could not find container \"1069ebf1855739d1a1f6c7ce2a5d28f1ff7cff857c56857f144232a223ffafa6\": container with ID starting with 1069ebf1855739d1a1f6c7ce2a5d28f1ff7cff857c56857f144232a223ffafa6 not found: ID does not exist" Aug 01 00:59:22.177795 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.177769 2563 scope.go:117] "RemoveContainer" containerID="f40146ebb63694919648dfeb2cf2182464b126aa8775a81d50dd4d4450b8fcc7" Aug 01 00:59:22.185301 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.185286 2563 scope.go:117] "RemoveContainer" containerID="f40146ebb63694919648dfeb2cf2182464b126aa8775a81d50dd4d4450b8fcc7" Aug 01 00:59:22.185541 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:59:22.185522 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40146ebb63694919648dfeb2cf2182464b126aa8775a81d50dd4d4450b8fcc7\": container with ID starting with f40146ebb63694919648dfeb2cf2182464b126aa8775a81d50dd4d4450b8fcc7 not found: ID does not exist" containerID="f40146ebb63694919648dfeb2cf2182464b126aa8775a81d50dd4d4450b8fcc7" Aug 01 00:59:22.185603 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.185545 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40146ebb63694919648dfeb2cf2182464b126aa8775a81d50dd4d4450b8fcc7"} err="failed to get container status \"f40146ebb63694919648dfeb2cf2182464b126aa8775a81d50dd4d4450b8fcc7\": rpc error: code = NotFound desc = could not find container \"f40146ebb63694919648dfeb2cf2182464b126aa8775a81d50dd4d4450b8fcc7\": container with ID starting with f40146ebb63694919648dfeb2cf2182464b126aa8775a81d50dd4d4450b8fcc7 not found: ID does not exist" Aug 01 00:59:22.190345 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.190212 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45"] Aug 01 00:59:22.191914 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.191894 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d5eeee24-56f494fdd8-4xn45"] Aug 01 00:59:22.205825 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.205803 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498"] Aug 01 00:59:22.214533 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.214505 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d5eeee24-6d49b55744-hw498"] Aug 01 00:59:22.229682 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.229653 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-istio-envoy\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.229682 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.229681 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-istio-data\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.229876 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.229699 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89jwb\" (UniqueName: \"kubernetes.io/projected/7088e54f-ecb7-4743-a345-d20230e59445-kube-api-access-89jwb\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.229876 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.229720 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7088e54f-ecb7-4743-a345-d20230e59445-istio-podinfo\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.229876 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.229741 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7088e54f-ecb7-4743-a345-d20230e59445-istio-token\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.230005 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.229869 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-credential-socket\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.230005 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.229904 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-workload-certs\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.230005 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.229969 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-workload-socket\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.230141 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.230007 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7088e54f-ecb7-4743-a345-d20230e59445-istiod-ca-cert\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.330665 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.330572 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-workload-socket\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.330665 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.330622 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7088e54f-ecb7-4743-a345-d20230e59445-istiod-ca-cert\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.330880 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.330732 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-istio-envoy\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.330880 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.330758 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-istio-data\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.330880 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.330778 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89jwb\" (UniqueName: \"kubernetes.io/projected/7088e54f-ecb7-4743-a345-d20230e59445-kube-api-access-89jwb\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.331041 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.330931 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7088e54f-ecb7-4743-a345-d20230e59445-istio-podinfo\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.331041 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.330960 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7088e54f-ecb7-4743-a345-d20230e59445-istio-token\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.331041 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.331018 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-workload-socket\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.331198 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.331046 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-credential-socket\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.331198 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.331095 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-workload-certs\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.331198 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.331122 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-istio-data\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.331399 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.331377 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-credential-socket\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.331552 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.331531 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7088e54f-ecb7-4743-a345-d20230e59445-istiod-ca-cert\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.331632 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.331603 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-workload-certs\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.333361 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.333341 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-istio-envoy\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.333449 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.333432 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7088e54f-ecb7-4743-a345-d20230e59445-istio-podinfo\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.338778 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.338749 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89jwb\" (UniqueName: \"kubernetes.io/projected/7088e54f-ecb7-4743-a345-d20230e59445-kube-api-access-89jwb\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.339163 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.339147 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7088e54f-ecb7-4743-a345-d20230e59445-istio-token\") pod \"e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.427482 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.427454 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:22.560349 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.560323 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp"] Aug 01 00:59:22.562263 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:59:22.562236 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7088e54f_ecb7_4743_a345_d20230e59445.slice/crio-c61389ba733431b469f3ce54c4b54d154b33f40ca315102eed4fc83803943866 WatchSource:0}: Error finding container c61389ba733431b469f3ce54c4b54d154b33f40ca315102eed4fc83803943866: Status 404 returned error can't find the container with id c61389ba733431b469f3ce54c4b54d154b33f40ca315102eed4fc83803943866 Aug 01 00:59:22.564264 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.564232 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:59:22.564345 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.564307 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:59:22.564408 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:22.564352 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:59:23.176511 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:23.176479 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" event={"ID":"7088e54f-ecb7-4743-a345-d20230e59445","Type":"ContainerStarted","Data":"b42345c4cd5d26265fc6d029b55140dd288565f3045bbab1bd7d5e54280461fa"} Aug 01 00:59:23.176945 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:23.176571 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" event={"ID":"7088e54f-ecb7-4743-a345-d20230e59445","Type":"ContainerStarted","Data":"c61389ba733431b469f3ce54c4b54d154b33f40ca315102eed4fc83803943866"} Aug 01 00:59:23.196823 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:23.196775 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" podStartSLOduration=1.196760445 podStartE2EDuration="1.196760445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:59:23.195142224 +0000 UTC m=+1851.983864051" watchObservedRunningTime="2026-08-01 00:59:23.196760445 +0000 UTC m=+1851.985482270" Aug 01 00:59:23.428050 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:23.427937 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:23.433265 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:23.433235 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:23.728314 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:23.728234 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f01e4d0-c115-4ea6-8551-dda951949990" path="/var/lib/kubelet/pods/3f01e4d0-c115-4ea6-8551-dda951949990/volumes" Aug 01 00:59:23.728712 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:23.728696 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6527228d-44f3-48a2-8153-78da9b4ce446" path="/var/lib/kubelet/pods/6527228d-44f3-48a2-8153-78da9b4ce446/volumes" Aug 01 00:59:24.182770 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:24.182732 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:24.183789 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:24.183768 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:25.117575 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:25.117535 2563 patch_prober.go:28] interesting pod/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Aug 01 00:59:25.117865 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:25.117596 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" podUID="3ad9be01-7425-485d-a2fd-312de890129c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 01 00:59:26.415921 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.415890 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh"] Aug 01 00:59:26.420776 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.420755 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" Aug 01 00:59:26.423566 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.423539 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-99a0aaf4-dockercfg-44zzb\"" Aug 01 00:59:26.424171 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.424150 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-99a0aaf4\"" Aug 01 00:59:26.430787 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.430763 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh"] Aug 01 00:59:26.571764 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.571729 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvmdb\" (UniqueName: \"kubernetes.io/projected/e3952d3f-aa1b-432b-a38e-c6fc6833446a-kube-api-access-hvmdb\") pod \"payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh\" (UID: \"e3952d3f-aa1b-432b-a38e-c6fc6833446a\") " pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" Aug 01 00:59:26.571954 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.571855 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3952d3f-aa1b-432b-a38e-c6fc6833446a-plugins-config-volume\") pod \"payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh\" (UID: \"e3952d3f-aa1b-432b-a38e-c6fc6833446a\") " pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" Aug 01 00:59:26.633093 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.633046 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:59:26.672749 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.672674 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmdb\" (UniqueName: \"kubernetes.io/projected/e3952d3f-aa1b-432b-a38e-c6fc6833446a-kube-api-access-hvmdb\") pod \"payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh\" (UID: \"e3952d3f-aa1b-432b-a38e-c6fc6833446a\") " pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" Aug 01 00:59:26.672885 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.672752 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3952d3f-aa1b-432b-a38e-c6fc6833446a-plugins-config-volume\") pod \"payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh\" (UID: \"e3952d3f-aa1b-432b-a38e-c6fc6833446a\") " pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" Aug 01 00:59:26.673439 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.673421 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3952d3f-aa1b-432b-a38e-c6fc6833446a-plugins-config-volume\") pod \"payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh\" (UID: \"e3952d3f-aa1b-432b-a38e-c6fc6833446a\") " pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" Aug 01 00:59:26.680330 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.680303 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvmdb\" (UniqueName: \"kubernetes.io/projected/e3952d3f-aa1b-432b-a38e-c6fc6833446a-kube-api-access-hvmdb\") pod \"payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh\" (UID: \"e3952d3f-aa1b-432b-a38e-c6fc6833446a\") " pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" Aug 01 00:59:26.733456 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.733423 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" Aug 01 00:59:26.773962 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.773926 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad9be01-7425-485d-a2fd-312de890129c-istio-token\") pod \"3ad9be01-7425-485d-a2fd-312de890129c\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " Aug 01 00:59:26.774151 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.773979 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-workload-socket\") pod \"3ad9be01-7425-485d-a2fd-312de890129c\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " Aug 01 00:59:26.774151 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.774009 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad9be01-7425-485d-a2fd-312de890129c-istio-podinfo\") pod \"3ad9be01-7425-485d-a2fd-312de890129c\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " Aug 01 00:59:26.774151 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.774039 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad9be01-7425-485d-a2fd-312de890129c-istiod-ca-cert\") pod \"3ad9be01-7425-485d-a2fd-312de890129c\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " Aug 01 00:59:26.774151 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.774111 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-credential-socket\") pod \"3ad9be01-7425-485d-a2fd-312de890129c\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " Aug 01 00:59:26.774381 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.774178 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfq55\" (UniqueName: \"kubernetes.io/projected/3ad9be01-7425-485d-a2fd-312de890129c-kube-api-access-gfq55\") pod \"3ad9be01-7425-485d-a2fd-312de890129c\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " Aug 01 00:59:26.774381 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.774206 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-istio-envoy\") pod \"3ad9be01-7425-485d-a2fd-312de890129c\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " Aug 01 00:59:26.774381 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.774232 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-istio-data\") pod \"3ad9be01-7425-485d-a2fd-312de890129c\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " Aug 01 00:59:26.774381 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.774254 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-workload-certs\") pod \"3ad9be01-7425-485d-a2fd-312de890129c\" (UID: \"3ad9be01-7425-485d-a2fd-312de890129c\") " Aug 01 00:59:26.774381 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.774342 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ad9be01-7425-485d-a2fd-312de890129c" (UID: "3ad9be01-7425-485d-a2fd-312de890129c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:59:26.774622 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.774586 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-workload-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:26.774735 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.774705 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad9be01-7425-485d-a2fd-312de890129c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ad9be01-7425-485d-a2fd-312de890129c" (UID: "3ad9be01-7425-485d-a2fd-312de890129c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:59:26.774869 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.774782 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ad9be01-7425-485d-a2fd-312de890129c" (UID: "3ad9be01-7425-485d-a2fd-312de890129c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:59:26.775022 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.775002 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ad9be01-7425-485d-a2fd-312de890129c" (UID: "3ad9be01-7425-485d-a2fd-312de890129c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:59:26.775268 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.775242 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ad9be01-7425-485d-a2fd-312de890129c" (UID: "3ad9be01-7425-485d-a2fd-312de890129c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:59:26.776377 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.776352 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad9be01-7425-485d-a2fd-312de890129c-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ad9be01-7425-485d-a2fd-312de890129c" (UID: "3ad9be01-7425-485d-a2fd-312de890129c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:59:26.776466 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.776425 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ad9be01-7425-485d-a2fd-312de890129c" (UID: "3ad9be01-7425-485d-a2fd-312de890129c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:59:26.776663 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.776649 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad9be01-7425-485d-a2fd-312de890129c-kube-api-access-gfq55" (OuterVolumeSpecName: "kube-api-access-gfq55") pod "3ad9be01-7425-485d-a2fd-312de890129c" (UID: "3ad9be01-7425-485d-a2fd-312de890129c"). InnerVolumeSpecName "kube-api-access-gfq55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:59:26.776901 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.776884 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ad9be01-7425-485d-a2fd-312de890129c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ad9be01-7425-485d-a2fd-312de890129c" (UID: "3ad9be01-7425-485d-a2fd-312de890129c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 01 00:59:26.872531 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.872506 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh"] Aug 01 00:59:26.874673 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:59:26.874643 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3952d3f_aa1b_432b_a38e_c6fc6833446a.slice/crio-813861f19329975349d3cb5183f0c7a033aefbf406e90a4a3beb91f6d4a9cdf7 WatchSource:0}: Error finding container 813861f19329975349d3cb5183f0c7a033aefbf406e90a4a3beb91f6d4a9cdf7: Status 404 returned error can't find the container with id 813861f19329975349d3cb5183f0c7a033aefbf406e90a4a3beb91f6d4a9cdf7 Aug 01 00:59:26.875080 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.875038 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad9be01-7425-485d-a2fd-312de890129c-istiod-ca-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:26.875080 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.875076 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-credential-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:26.875247 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.875089 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfq55\" (UniqueName: \"kubernetes.io/projected/3ad9be01-7425-485d-a2fd-312de890129c-kube-api-access-gfq55\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:26.875247 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.875099 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-istio-envoy\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:26.875247 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.875107 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-istio-data\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:26.875247 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.875116 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad9be01-7425-485d-a2fd-312de890129c-workload-certs\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:26.875247 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.875124 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad9be01-7425-485d-a2fd-312de890129c-istio-token\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:26.875247 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:26.875131 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad9be01-7425-485d-a2fd-312de890129c-istio-podinfo\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:27.199922 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.199820 2563 generic.go:356] "Generic (PLEG): container finished" podID="3ad9be01-7425-485d-a2fd-312de890129c" containerID="b80ff522770823dabf4791931f73ea698ccb008477511c742b8054f14cdf9124" exitCode=0 Aug 01 00:59:27.199922 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.199877 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" Aug 01 00:59:27.199922 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.199900 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" event={"ID":"3ad9be01-7425-485d-a2fd-312de890129c","Type":"ContainerDied","Data":"b80ff522770823dabf4791931f73ea698ccb008477511c742b8054f14cdf9124"} Aug 01 00:59:27.200236 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.199965 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt" event={"ID":"3ad9be01-7425-485d-a2fd-312de890129c","Type":"ContainerDied","Data":"f229ce746d1ce0ded70435d3b0cf0de43d1f7b5d1df441e63eba662148946409"} Aug 01 00:59:27.200236 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.200001 2563 scope.go:117] "RemoveContainer" containerID="b80ff522770823dabf4791931f73ea698ccb008477511c742b8054f14cdf9124" Aug 01 00:59:27.201647 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.201621 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" event={"ID":"e3952d3f-aa1b-432b-a38e-c6fc6833446a","Type":"ContainerStarted","Data":"c3a4ca9333730e4f4589bb39703e485389a65c15e1a66eb2f0ce65ede4202fde"} Aug 01 00:59:27.201788 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.201666 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" event={"ID":"e3952d3f-aa1b-432b-a38e-c6fc6833446a","Type":"ContainerStarted","Data":"813861f19329975349d3cb5183f0c7a033aefbf406e90a4a3beb91f6d4a9cdf7"} Aug 01 00:59:27.201788 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.201692 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" Aug 01 00:59:27.209268 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.209250 2563 scope.go:117] "RemoveContainer" containerID="f638bf7e86b996c2a0a136f32fe79138a52edc9c65fb49f89ee9cca1a38b3ec1" Aug 01 00:59:27.217924 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.217873 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" podStartSLOduration=1.217858121 podStartE2EDuration="1.217858121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:59:27.216816 +0000 UTC m=+1856.005537872" watchObservedRunningTime="2026-08-01 00:59:27.217858121 +0000 UTC m=+1856.006579945" Aug 01 00:59:27.218927 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.218887 2563 scope.go:117] "RemoveContainer" containerID="b80ff522770823dabf4791931f73ea698ccb008477511c742b8054f14cdf9124" Aug 01 00:59:27.219283 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:59:27.219251 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80ff522770823dabf4791931f73ea698ccb008477511c742b8054f14cdf9124\": container with ID starting with b80ff522770823dabf4791931f73ea698ccb008477511c742b8054f14cdf9124 not found: ID does not exist" containerID="b80ff522770823dabf4791931f73ea698ccb008477511c742b8054f14cdf9124" Aug 01 00:59:27.219425 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.219296 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80ff522770823dabf4791931f73ea698ccb008477511c742b8054f14cdf9124"} err="failed to get container status \"b80ff522770823dabf4791931f73ea698ccb008477511c742b8054f14cdf9124\": rpc error: code = NotFound desc = could not find container \"b80ff522770823dabf4791931f73ea698ccb008477511c742b8054f14cdf9124\": container with ID starting with b80ff522770823dabf4791931f73ea698ccb008477511c742b8054f14cdf9124 not found: ID does not exist" Aug 01 00:59:27.219425 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.219321 2563 scope.go:117] "RemoveContainer" containerID="f638bf7e86b996c2a0a136f32fe79138a52edc9c65fb49f89ee9cca1a38b3ec1" Aug 01 00:59:27.219635 ip-10-0-128-195 kubenswrapper[2563]: E0801 00:59:27.219614 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f638bf7e86b996c2a0a136f32fe79138a52edc9c65fb49f89ee9cca1a38b3ec1\": container with ID starting with f638bf7e86b996c2a0a136f32fe79138a52edc9c65fb49f89ee9cca1a38b3ec1 not found: ID does not exist" containerID="f638bf7e86b996c2a0a136f32fe79138a52edc9c65fb49f89ee9cca1a38b3ec1" Aug 01 00:59:27.219694 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.219644 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f638bf7e86b996c2a0a136f32fe79138a52edc9c65fb49f89ee9cca1a38b3ec1"} err="failed to get container status \"f638bf7e86b996c2a0a136f32fe79138a52edc9c65fb49f89ee9cca1a38b3ec1\": rpc error: code = NotFound desc = could not find container \"f638bf7e86b996c2a0a136f32fe79138a52edc9c65fb49f89ee9cca1a38b3ec1\": container with ID starting with f638bf7e86b996c2a0a136f32fe79138a52edc9c65fb49f89ee9cca1a38b3ec1 not found: ID does not exist" Aug 01 00:59:27.230210 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.230190 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt"] Aug 01 00:59:27.233705 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.233685 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d5eeee24-gw-openshift-default-9895fb5b-gxkkt"] Aug 01 00:59:27.730872 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:27.730786 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ad9be01-7425-485d-a2fd-312de890129c" path="/var/lib/kubelet/pods/3ad9be01-7425-485d-a2fd-312de890129c/volumes" Aug 01 00:59:38.211450 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:38.211414 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" Aug 01 00:59:38.277322 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:38.277289 2563 generic.go:356] "Generic (PLEG): container finished" podID="7088e54f-ecb7-4743-a345-d20230e59445" containerID="b42345c4cd5d26265fc6d029b55140dd288565f3045bbab1bd7d5e54280461fa" exitCode=0 Aug 01 00:59:38.277499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:38.277341 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" event={"ID":"7088e54f-ecb7-4743-a345-d20230e59445","Type":"ContainerDied","Data":"b42345c4cd5d26265fc6d029b55140dd288565f3045bbab1bd7d5e54280461fa"} Aug 01 00:59:38.277800 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:38.277781 2563 scope.go:117] "RemoveContainer" containerID="b42345c4cd5d26265fc6d029b55140dd288565f3045bbab1bd7d5e54280461fa" Aug 01 00:59:38.279000 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:38.278966 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:59:38.279091 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:38.279073 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:59:38.279147 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:38.279130 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:59:39.183686 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:39.183651 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:39.284512 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:39.284482 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" event={"ID":"7088e54f-ecb7-4743-a345-d20230e59445","Type":"ContainerStarted","Data":"4681209c9fb447802be43c1ec26763ed0ec723b87391e2e15d717b1f4b8ea698"} Aug 01 00:59:39.428015 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:39.427983 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:39.429225 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:39.429194 2563 patch_prober.go:28] interesting pod/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 01 00:59:39.429422 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:39.429397 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" podUID="7088e54f-ecb7-4743-a345-d20230e59445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 01 00:59:40.428392 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:40.428344 2563 patch_prober.go:28] interesting pod/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 01 00:59:40.428905 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:40.428415 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" podUID="7088e54f-ecb7-4743-a345-d20230e59445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 01 00:59:41.428305 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:41.428263 2563 patch_prober.go:28] interesting pod/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 01 00:59:41.428598 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:41.428571 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" podUID="7088e54f-ecb7-4743-a345-d20230e59445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 01 00:59:42.428616 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:42.428572 2563 patch_prober.go:28] interesting pod/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 01 00:59:42.429149 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:42.428658 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" podUID="7088e54f-ecb7-4743-a345-d20230e59445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 01 00:59:43.428898 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:43.428859 2563 patch_prober.go:28] interesting pod/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 01 00:59:43.429489 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:43.429455 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" podUID="7088e54f-ecb7-4743-a345-d20230e59445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 01 00:59:44.432412 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:44.432383 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:44.432812 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:44.432657 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:44.433387 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:44.433370 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 00:59:56.190931 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:56.190832 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh"] Aug 01 00:59:56.191504 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:56.191366 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" podUID="e3952d3f-aa1b-432b-a38e-c6fc6833446a" containerName="payload-processing" containerID="cri-o://c3a4ca9333730e4f4589bb39703e485389a65c15e1a66eb2f0ce65ede4202fde" gracePeriod=30 Aug 01 00:59:56.374634 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:56.374598 2563 generic.go:356] "Generic (PLEG): container finished" podID="e3952d3f-aa1b-432b-a38e-c6fc6833446a" containerID="c3a4ca9333730e4f4589bb39703e485389a65c15e1a66eb2f0ce65ede4202fde" exitCode=0 Aug 01 00:59:56.374791 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:56.374677 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" event={"ID":"e3952d3f-aa1b-432b-a38e-c6fc6833446a","Type":"ContainerDied","Data":"c3a4ca9333730e4f4589bb39703e485389a65c15e1a66eb2f0ce65ede4202fde"} Aug 01 00:59:56.457587 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:56.457551 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" Aug 01 00:59:56.561494 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:56.561460 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvmdb\" (UniqueName: \"kubernetes.io/projected/e3952d3f-aa1b-432b-a38e-c6fc6833446a-kube-api-access-hvmdb\") pod \"e3952d3f-aa1b-432b-a38e-c6fc6833446a\" (UID: \"e3952d3f-aa1b-432b-a38e-c6fc6833446a\") " Aug 01 00:59:56.561651 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:56.561568 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3952d3f-aa1b-432b-a38e-c6fc6833446a-plugins-config-volume\") pod \"e3952d3f-aa1b-432b-a38e-c6fc6833446a\" (UID: \"e3952d3f-aa1b-432b-a38e-c6fc6833446a\") " Aug 01 00:59:56.561924 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:56.561902 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3952d3f-aa1b-432b-a38e-c6fc6833446a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e3952d3f-aa1b-432b-a38e-c6fc6833446a" (UID: "e3952d3f-aa1b-432b-a38e-c6fc6833446a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:59:56.563496 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:56.563471 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3952d3f-aa1b-432b-a38e-c6fc6833446a-kube-api-access-hvmdb" (OuterVolumeSpecName: "kube-api-access-hvmdb") pod "e3952d3f-aa1b-432b-a38e-c6fc6833446a" (UID: "e3952d3f-aa1b-432b-a38e-c6fc6833446a"). InnerVolumeSpecName "kube-api-access-hvmdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:59:56.662404 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:56.662370 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvmdb\" (UniqueName: \"kubernetes.io/projected/e3952d3f-aa1b-432b-a38e-c6fc6833446a-kube-api-access-hvmdb\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:56.662404 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:56.662400 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3952d3f-aa1b-432b-a38e-c6fc6833446a-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 00:59:57.380488 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:57.380461 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" Aug 01 00:59:57.380907 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:57.380496 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh" event={"ID":"e3952d3f-aa1b-432b-a38e-c6fc6833446a","Type":"ContainerDied","Data":"813861f19329975349d3cb5183f0c7a033aefbf406e90a4a3beb91f6d4a9cdf7"} Aug 01 00:59:57.380907 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:57.380554 2563 scope.go:117] "RemoveContainer" containerID="c3a4ca9333730e4f4589bb39703e485389a65c15e1a66eb2f0ce65ede4202fde" Aug 01 00:59:57.406813 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:57.406776 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh"] Aug 01 00:59:57.409615 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:57.409587 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-99a0aaf4-8474778fbb-xmflh"] Aug 01 00:59:57.433085 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:57.433042 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp"] Aug 01 00:59:57.433305 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:57.433268 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" podUID="7088e54f-ecb7-4743-a345-d20230e59445" containerName="istio-proxy" containerID="cri-o://4681209c9fb447802be43c1ec26763ed0ec723b87391e2e15d717b1f4b8ea698" gracePeriod=30 Aug 01 00:59:57.729195 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:57.729115 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3952d3f-aa1b-432b-a38e-c6fc6833446a" path="/var/lib/kubelet/pods/e3952d3f-aa1b-432b-a38e-c6fc6833446a/volumes" Aug 01 00:59:58.129490 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.129452 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts"] Aug 01 00:59:58.129994 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.129976 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3952d3f-aa1b-432b-a38e-c6fc6833446a" containerName="payload-processing" Aug 01 00:59:58.129994 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.129995 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3952d3f-aa1b-432b-a38e-c6fc6833446a" containerName="payload-processing" Aug 01 00:59:58.130172 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.130015 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ad9be01-7425-485d-a2fd-312de890129c" containerName="istio-proxy" Aug 01 00:59:58.130172 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.130024 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ad9be01-7425-485d-a2fd-312de890129c" containerName="istio-proxy" Aug 01 00:59:58.130172 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.130047 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ad9be01-7425-485d-a2fd-312de890129c" containerName="istio-proxy" Aug 01 00:59:58.130172 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.130103 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ad9be01-7425-485d-a2fd-312de890129c" containerName="istio-proxy" Aug 01 00:59:58.130392 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.130228 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ad9be01-7425-485d-a2fd-312de890129c" containerName="istio-proxy" Aug 01 00:59:58.130392 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.130241 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3952d3f-aa1b-432b-a38e-c6fc6833446a" containerName="payload-processing" Aug 01 00:59:58.138346 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.138229 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.141939 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.141912 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c7aeac35-openshift-default-dockercfg-8mxrk\"" Aug 01 00:59:58.144531 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.144506 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts"] Aug 01 00:59:58.275499 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.275464 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x997\" (UniqueName: \"kubernetes.io/projected/50f30b3b-6071-4ef7-9ea9-a8a70b174113-kube-api-access-5x997\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.275651 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.275509 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-data\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.275651 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.275565 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istiod-ca-cert\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.275651 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.275644 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-credential-socket\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.275787 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.275665 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-token\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.275787 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.275692 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-workload-certs\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.275787 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.275716 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-workload-socket\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.275787 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.275774 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-envoy\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.275929 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.275815 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-podinfo\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.376446 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.376409 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-credential-socket\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.376446 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.376447 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-token\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.376682 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.376471 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-workload-certs\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.376682 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.376497 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-workload-socket\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.376682 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.376516 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-envoy\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.376682 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.376534 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-podinfo\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.376682 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.376557 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x997\" (UniqueName: \"kubernetes.io/projected/50f30b3b-6071-4ef7-9ea9-a8a70b174113-kube-api-access-5x997\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.376682 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.376595 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-data\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.376682 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.376628 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istiod-ca-cert\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.377052 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.376931 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-credential-socket\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.377052 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.377020 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-data\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.377181 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.377098 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-workload-certs\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.377181 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.377130 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-workload-socket\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.377287 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.377271 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istiod-ca-cert\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.379070 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.379036 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-envoy\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.379259 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.379242 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-podinfo\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.385496 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.385441 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-token\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.385829 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.385647 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x997\" (UniqueName: \"kubernetes.io/projected/50f30b3b-6071-4ef7-9ea9-a8a70b174113-kube-api-access-5x997\") pod \"e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.451193 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.451162 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:58.586558 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.586534 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts"] Aug 01 00:59:58.588513 ip-10-0-128-195 kubenswrapper[2563]: W0801 00:59:58.588481 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f30b3b_6071_4ef7_9ea9_a8a70b174113.slice/crio-f6608ef33d39b218ca2a0a2f77a7e0ed438ab72da9dcb60c9528c4e9d1987c34 WatchSource:0}: Error finding container f6608ef33d39b218ca2a0a2f77a7e0ed438ab72da9dcb60c9528c4e9d1987c34: Status 404 returned error can't find the container with id f6608ef33d39b218ca2a0a2f77a7e0ed438ab72da9dcb60c9528c4e9d1987c34 Aug 01 00:59:58.590479 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.590445 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:59:58.590572 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.590511 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:59:58.590572 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:58.590540 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 00:59:59.392272 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:59.392236 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" event={"ID":"50f30b3b-6071-4ef7-9ea9-a8a70b174113","Type":"ContainerStarted","Data":"192e77dade2949b6429c63546550296b75ae7bfd0bbd4bf9195fdd182a54b267"} Aug 01 00:59:59.392735 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:59.392296 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" event={"ID":"50f30b3b-6071-4ef7-9ea9-a8a70b174113","Type":"ContainerStarted","Data":"f6608ef33d39b218ca2a0a2f77a7e0ed438ab72da9dcb60c9528c4e9d1987c34"} Aug 01 00:59:59.418598 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:59.418549 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" podStartSLOduration=1.418531859 podStartE2EDuration="1.418531859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:59:59.416515379 +0000 UTC m=+1888.205237217" watchObservedRunningTime="2026-08-01 00:59:59.418531859 +0000 UTC m=+1888.207253684" Aug 01 00:59:59.435612 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:59.435576 2563 patch_prober.go:28] interesting pod/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 01 00:59:59.435743 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:59.435634 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" podUID="7088e54f-ecb7-4743-a345-d20230e59445" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 01 00:59:59.451444 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:59.451418 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 00:59:59.456231 ip-10-0-128-195 kubenswrapper[2563]: I0801 00:59:59.456206 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 01:00:00.397750 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:00.397721 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 01:00:00.398848 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:00.398824 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 01:00:01.847874 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:01.847838 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd"] Aug 01 01:00:01.848488 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:01.848467 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ad9be01-7425-485d-a2fd-312de890129c" containerName="istio-proxy" Aug 01 01:00:01.851736 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:01.851713 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:01.854610 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:01.854573 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 01 01:00:01.854782 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:01.854761 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:00:01.855739 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:01.855714 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c7aeac35-serving-cert\"" Aug 01 01:00:01.855853 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:01.855722 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gf2hx\"" Aug 01 01:00:01.871030 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:01.870999 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd"] Aug 01 01:00:01.910443 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:01.910410 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-maas-api-tls\") pod \"maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd\" (UID: \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:01.910586 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:01.910544 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chspl\" (UniqueName: \"kubernetes.io/projected/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-kube-api-access-chspl\") pod \"maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd\" (UID: \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:01.910647 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:01.910587 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-openshift-service-ca\") pod \"maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd\" (UID: \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:02.011492 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.011450 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-maas-api-tls\") pod \"maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd\" (UID: \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:02.011704 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.011547 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chspl\" (UniqueName: \"kubernetes.io/projected/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-kube-api-access-chspl\") pod \"maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd\" (UID: \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:02.011704 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.011580 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-openshift-service-ca\") pod \"maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd\" (UID: \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:02.012274 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.012249 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-openshift-service-ca\") pod \"maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd\" (UID: \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:02.013878 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.013847 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-maas-api-tls\") pod \"maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd\" (UID: \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:02.026606 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.026560 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chspl\" (UniqueName: \"kubernetes.io/projected/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-kube-api-access-chspl\") pod \"maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd\" (UID: \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:02.164859 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.164758 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:02.310194 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.310168 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd"] Aug 01 01:00:02.312500 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:00:02.312466 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2bad7cf_6886_4b31_8d89_3eceb8f24b5c.slice/crio-797ab3f7efecad7504ca3ff12aa7c2061690524a41ed0896eccee63f6151273c WatchSource:0}: Error finding container 797ab3f7efecad7504ca3ff12aa7c2061690524a41ed0896eccee63f6151273c: Status 404 returned error can't find the container with id 797ab3f7efecad7504ca3ff12aa7c2061690524a41ed0896eccee63f6151273c Aug 01 01:00:02.408233 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.408196 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" event={"ID":"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c","Type":"ContainerStarted","Data":"797ab3f7efecad7504ca3ff12aa7c2061690524a41ed0896eccee63f6151273c"} Aug 01 01:00:02.674610 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.674579 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 01:00:02.820092 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820049 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-workload-certs\") pod \"7088e54f-ecb7-4743-a345-d20230e59445\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " Aug 01 01:00:02.820237 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820108 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-workload-socket\") pod \"7088e54f-ecb7-4743-a345-d20230e59445\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " Aug 01 01:00:02.820237 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820177 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7088e54f-ecb7-4743-a345-d20230e59445-istiod-ca-cert\") pod \"7088e54f-ecb7-4743-a345-d20230e59445\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " Aug 01 01:00:02.820237 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820224 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89jwb\" (UniqueName: \"kubernetes.io/projected/7088e54f-ecb7-4743-a345-d20230e59445-kube-api-access-89jwb\") pod \"7088e54f-ecb7-4743-a345-d20230e59445\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " Aug 01 01:00:02.820401 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820264 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7088e54f-ecb7-4743-a345-d20230e59445-istio-podinfo\") pod \"7088e54f-ecb7-4743-a345-d20230e59445\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " Aug 01 01:00:02.820401 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820304 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-istio-data\") pod \"7088e54f-ecb7-4743-a345-d20230e59445\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " Aug 01 01:00:02.820401 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820302 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7088e54f-ecb7-4743-a345-d20230e59445" (UID: "7088e54f-ecb7-4743-a345-d20230e59445"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:00:02.820401 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820347 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7088e54f-ecb7-4743-a345-d20230e59445-istio-token\") pod \"7088e54f-ecb7-4743-a345-d20230e59445\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " Aug 01 01:00:02.820401 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820375 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7088e54f-ecb7-4743-a345-d20230e59445" (UID: "7088e54f-ecb7-4743-a345-d20230e59445"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:00:02.820401 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820384 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-istio-envoy\") pod \"7088e54f-ecb7-4743-a345-d20230e59445\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " Aug 01 01:00:02.820713 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820464 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-credential-socket\") pod \"7088e54f-ecb7-4743-a345-d20230e59445\" (UID: \"7088e54f-ecb7-4743-a345-d20230e59445\") " Aug 01 01:00:02.820713 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820534 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7088e54f-ecb7-4743-a345-d20230e59445-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7088e54f-ecb7-4743-a345-d20230e59445" (UID: "7088e54f-ecb7-4743-a345-d20230e59445"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:00:02.820826 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820776 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-istio-data" (OuterVolumeSpecName: "istio-data") pod "7088e54f-ecb7-4743-a345-d20230e59445" (UID: "7088e54f-ecb7-4743-a345-d20230e59445"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:00:02.820880 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820820 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-workload-certs\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:02.820880 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820839 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-workload-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:02.820880 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.820854 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7088e54f-ecb7-4743-a345-d20230e59445-istiod-ca-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:02.821018 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.821004 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7088e54f-ecb7-4743-a345-d20230e59445" (UID: "7088e54f-ecb7-4743-a345-d20230e59445"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:00:02.822775 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.822746 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7088e54f-ecb7-4743-a345-d20230e59445" (UID: "7088e54f-ecb7-4743-a345-d20230e59445"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:00:02.822963 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.822941 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7088e54f-ecb7-4743-a345-d20230e59445-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7088e54f-ecb7-4743-a345-d20230e59445" (UID: "7088e54f-ecb7-4743-a345-d20230e59445"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 01 01:00:02.822963 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.822952 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7088e54f-ecb7-4743-a345-d20230e59445-kube-api-access-89jwb" (OuterVolumeSpecName: "kube-api-access-89jwb") pod "7088e54f-ecb7-4743-a345-d20230e59445" (UID: "7088e54f-ecb7-4743-a345-d20230e59445"). InnerVolumeSpecName "kube-api-access-89jwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:00:02.823905 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.823883 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7088e54f-ecb7-4743-a345-d20230e59445-istio-token" (OuterVolumeSpecName: "istio-token") pod "7088e54f-ecb7-4743-a345-d20230e59445" (UID: "7088e54f-ecb7-4743-a345-d20230e59445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:00:02.922009 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.921974 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89jwb\" (UniqueName: \"kubernetes.io/projected/7088e54f-ecb7-4743-a345-d20230e59445-kube-api-access-89jwb\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:02.922009 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.922006 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7088e54f-ecb7-4743-a345-d20230e59445-istio-podinfo\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:02.922009 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.922016 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-istio-data\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:02.922450 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.922024 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7088e54f-ecb7-4743-a345-d20230e59445-istio-token\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:02.922450 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.922032 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-istio-envoy\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:02.922450 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:02.922040 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7088e54f-ecb7-4743-a345-d20230e59445-credential-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:03.415919 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.415879 2563 generic.go:356] "Generic (PLEG): container finished" podID="7088e54f-ecb7-4743-a345-d20230e59445" containerID="4681209c9fb447802be43c1ec26763ed0ec723b87391e2e15d717b1f4b8ea698" exitCode=0 Aug 01 01:00:03.416132 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.415926 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" Aug 01 01:00:03.416132 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.415963 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" event={"ID":"7088e54f-ecb7-4743-a345-d20230e59445","Type":"ContainerDied","Data":"4681209c9fb447802be43c1ec26763ed0ec723b87391e2e15d717b1f4b8ea698"} Aug 01 01:00:03.416132 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.416024 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp" event={"ID":"7088e54f-ecb7-4743-a345-d20230e59445","Type":"ContainerDied","Data":"c61389ba733431b469f3ce54c4b54d154b33f40ca315102eed4fc83803943866"} Aug 01 01:00:03.416132 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.416050 2563 scope.go:117] "RemoveContainer" containerID="4681209c9fb447802be43c1ec26763ed0ec723b87391e2e15d717b1f4b8ea698" Aug 01 01:00:03.417764 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.417740 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" event={"ID":"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c","Type":"ContainerStarted","Data":"d31ed5913949d10d9ca84ce7cada3c1957d947eb9e98cf0655699be63bb8533b"} Aug 01 01:00:03.417889 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.417794 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:03.425483 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.425459 2563 scope.go:117] "RemoveContainer" containerID="b42345c4cd5d26265fc6d029b55140dd288565f3045bbab1bd7d5e54280461fa" Aug 01 01:00:03.433638 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.433619 2563 scope.go:117] "RemoveContainer" containerID="4681209c9fb447802be43c1ec26763ed0ec723b87391e2e15d717b1f4b8ea698" Aug 01 01:00:03.433919 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:00:03.433899 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4681209c9fb447802be43c1ec26763ed0ec723b87391e2e15d717b1f4b8ea698\": container with ID starting with 4681209c9fb447802be43c1ec26763ed0ec723b87391e2e15d717b1f4b8ea698 not found: ID does not exist" containerID="4681209c9fb447802be43c1ec26763ed0ec723b87391e2e15d717b1f4b8ea698" Aug 01 01:00:03.433972 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.433930 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4681209c9fb447802be43c1ec26763ed0ec723b87391e2e15d717b1f4b8ea698"} err="failed to get container status \"4681209c9fb447802be43c1ec26763ed0ec723b87391e2e15d717b1f4b8ea698\": rpc error: code = NotFound desc = could not find container \"4681209c9fb447802be43c1ec26763ed0ec723b87391e2e15d717b1f4b8ea698\": container with ID starting with 4681209c9fb447802be43c1ec26763ed0ec723b87391e2e15d717b1f4b8ea698 not found: ID does not exist" Aug 01 01:00:03.433972 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.433948 2563 scope.go:117] "RemoveContainer" containerID="b42345c4cd5d26265fc6d029b55140dd288565f3045bbab1bd7d5e54280461fa" Aug 01 01:00:03.434218 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:00:03.434198 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b42345c4cd5d26265fc6d029b55140dd288565f3045bbab1bd7d5e54280461fa\": container with ID starting with b42345c4cd5d26265fc6d029b55140dd288565f3045bbab1bd7d5e54280461fa not found: ID does not exist" containerID="b42345c4cd5d26265fc6d029b55140dd288565f3045bbab1bd7d5e54280461fa" Aug 01 01:00:03.434320 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.434223 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b42345c4cd5d26265fc6d029b55140dd288565f3045bbab1bd7d5e54280461fa"} err="failed to get container status \"b42345c4cd5d26265fc6d029b55140dd288565f3045bbab1bd7d5e54280461fa\": rpc error: code = NotFound desc = could not find container \"b42345c4cd5d26265fc6d029b55140dd288565f3045bbab1bd7d5e54280461fa\": container with ID starting with b42345c4cd5d26265fc6d029b55140dd288565f3045bbab1bd7d5e54280461fa not found: ID does not exist" Aug 01 01:00:03.442679 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.442639 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" podStartSLOduration=1.949509984 podStartE2EDuration="2.442624448s" totalImagesPullingTime="493.114464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:00:03.441339307 +0000 UTC m=+1892.230061157" watchObservedRunningTime="2026-08-01 01:00:03.442624448 +0000 UTC m=+1892.231346317" Aug 01 01:00:03.461413 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.461386 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp"] Aug 01 01:00:03.468207 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.468184 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-99a0aaf4-openshift-default-7b6f69f55d-5fpcp"] Aug 01 01:00:03.728763 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:03.728679 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7088e54f-ecb7-4743-a345-d20230e59445" path="/var/lib/kubelet/pods/7088e54f-ecb7-4743-a345-d20230e59445/volumes" Aug 01 01:00:09.436714 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:09.436675 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:15.399304 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:15.399261 2563 patch_prober.go:28] interesting pod/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Aug 01 01:00:15.399798 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:15.399329 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Aug 01 01:00:15.484302 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:15.484267 2563 generic.go:356] "Generic (PLEG): container finished" podID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerID="192e77dade2949b6429c63546550296b75ae7bfd0bbd4bf9195fdd182a54b267" exitCode=0 Aug 01 01:00:15.484462 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:15.484314 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" event={"ID":"50f30b3b-6071-4ef7-9ea9-a8a70b174113","Type":"ContainerDied","Data":"192e77dade2949b6429c63546550296b75ae7bfd0bbd4bf9195fdd182a54b267"} Aug 01 01:00:15.484755 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:15.484742 2563 scope.go:117] "RemoveContainer" containerID="192e77dade2949b6429c63546550296b75ae7bfd0bbd4bf9195fdd182a54b267" Aug 01 01:00:15.485742 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:15.485709 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:00:15.485842 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:15.485787 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:00:15.485842 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:15.485825 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:00:16.490987 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:16.490953 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" event={"ID":"50f30b3b-6071-4ef7-9ea9-a8a70b174113","Type":"ContainerStarted","Data":"ec6cf9ecabd0f3cca86644e21a5a670eadbb9068963eb228dca12e1a8cbaea2b"} Aug 01 01:00:17.452251 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:17.452204 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 01:00:17.452465 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:17.452440 2563 patch_prober.go:28] interesting pod/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Aug 01 01:00:17.452540 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:17.452511 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Aug 01 01:00:18.451938 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:18.451900 2563 patch_prober.go:28] interesting pod/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Aug 01 01:00:18.452355 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:18.451968 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Aug 01 01:00:19.452099 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:19.452001 2563 patch_prober.go:28] interesting pod/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Aug 01 01:00:19.452099 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:19.452079 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Aug 01 01:00:20.451952 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:20.451915 2563 patch_prober.go:28] interesting pod/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Aug 01 01:00:20.452165 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:20.451974 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Aug 01 01:00:21.452341 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:21.452295 2563 patch_prober.go:28] interesting pod/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Aug 01 01:00:21.452697 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:21.452371 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Aug 01 01:00:22.455672 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:22.455638 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 01:00:22.456083 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:22.455893 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 01:00:22.456573 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:22.456554 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 01:00:32.315158 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.315123 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd"] Aug 01 01:00:32.316774 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.315346 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" podUID="f2bad7cf-6886-4b31-8d89-3eceb8f24b5c" containerName="maas-api" containerID="cri-o://d31ed5913949d10d9ca84ce7cada3c1957d947eb9e98cf0655699be63bb8533b" gracePeriod=30 Aug 01 01:00:32.571644 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.571543 2563 generic.go:356] "Generic (PLEG): container finished" podID="f2bad7cf-6886-4b31-8d89-3eceb8f24b5c" containerID="d31ed5913949d10d9ca84ce7cada3c1957d947eb9e98cf0655699be63bb8533b" exitCode=0 Aug 01 01:00:32.571644 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.571614 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" event={"ID":"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c","Type":"ContainerDied","Data":"d31ed5913949d10d9ca84ce7cada3c1957d947eb9e98cf0655699be63bb8533b"} Aug 01 01:00:32.571834 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.571664 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" event={"ID":"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c","Type":"ContainerDied","Data":"797ab3f7efecad7504ca3ff12aa7c2061690524a41ed0896eccee63f6151273c"} Aug 01 01:00:32.571834 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.571680 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="797ab3f7efecad7504ca3ff12aa7c2061690524a41ed0896eccee63f6151273c" Aug 01 01:00:32.576896 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.576875 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:32.612542 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.612501 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chspl\" (UniqueName: \"kubernetes.io/projected/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-kube-api-access-chspl\") pod \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\" (UID: \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\") " Aug 01 01:00:32.615019 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.612605 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-openshift-service-ca\") pod \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\" (UID: \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\") " Aug 01 01:00:32.615019 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.612657 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-maas-api-tls\") pod \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\" (UID: \"f2bad7cf-6886-4b31-8d89-3eceb8f24b5c\") " Aug 01 01:00:32.615019 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.613151 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f2bad7cf-6886-4b31-8d89-3eceb8f24b5c" (UID: "f2bad7cf-6886-4b31-8d89-3eceb8f24b5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:00:32.615488 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.615463 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f2bad7cf-6886-4b31-8d89-3eceb8f24b5c" (UID: "f2bad7cf-6886-4b31-8d89-3eceb8f24b5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:00:32.616266 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.616230 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-kube-api-access-chspl" (OuterVolumeSpecName: "kube-api-access-chspl") pod "f2bad7cf-6886-4b31-8d89-3eceb8f24b5c" (UID: "f2bad7cf-6886-4b31-8d89-3eceb8f24b5c"). InnerVolumeSpecName "kube-api-access-chspl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:00:32.658996 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.658953 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts"] Aug 01 01:00:32.659274 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.659230 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" containerID="cri-o://ec6cf9ecabd0f3cca86644e21a5a670eadbb9068963eb228dca12e1a8cbaea2b" gracePeriod=30 Aug 01 01:00:32.714413 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.714375 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-maas-api-tls\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:32.714413 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.714412 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chspl\" (UniqueName: \"kubernetes.io/projected/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-kube-api-access-chspl\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:32.714605 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:32.714426 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c-openshift-service-ca\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:33.576736 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:33.576702 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd" Aug 01 01:00:33.604286 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:33.604258 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd"] Aug 01 01:00:33.610903 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:33.610880 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7aeac35-98b9b9675-grqnd"] Aug 01 01:00:33.727928 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:33.727896 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2bad7cf-6886-4b31-8d89-3eceb8f24b5c" path="/var/lib/kubelet/pods/f2bad7cf-6886-4b31-8d89-3eceb8f24b5c/volumes" Aug 01 01:00:37.458559 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.458514 2563 patch_prober.go:28] interesting pod/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Aug 01 01:00:37.458967 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.458577 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 01 01:00:37.917521 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.917496 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 01:00:37.965347 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.965265 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-envoy\") pod \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " Aug 01 01:00:37.965347 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.965326 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istiod-ca-cert\") pod \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " Aug 01 01:00:37.965578 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.965380 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-podinfo\") pod \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " Aug 01 01:00:37.965578 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.965436 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-workload-certs\") pod \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " Aug 01 01:00:37.965578 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.965467 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-credential-socket\") pod \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " Aug 01 01:00:37.965578 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.965534 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-data\") pod \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " Aug 01 01:00:37.965578 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.965560 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-token\") pod \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " Aug 01 01:00:37.965864 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.965591 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-workload-socket\") pod \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " Aug 01 01:00:37.965864 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.965628 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x997\" (UniqueName: \"kubernetes.io/projected/50f30b3b-6071-4ef7-9ea9-a8a70b174113-kube-api-access-5x997\") pod \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\" (UID: \"50f30b3b-6071-4ef7-9ea9-a8a70b174113\") " Aug 01 01:00:37.965864 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.965729 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "50f30b3b-6071-4ef7-9ea9-a8a70b174113" (UID: "50f30b3b-6071-4ef7-9ea9-a8a70b174113"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:00:37.965864 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.965749 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "50f30b3b-6071-4ef7-9ea9-a8a70b174113" (UID: "50f30b3b-6071-4ef7-9ea9-a8a70b174113"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:00:37.966111 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.965961 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "50f30b3b-6071-4ef7-9ea9-a8a70b174113" (UID: "50f30b3b-6071-4ef7-9ea9-a8a70b174113"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:00:37.966111 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.965985 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-workload-certs\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:37.966111 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.966004 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istiod-ca-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:37.966298 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.966132 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "50f30b3b-6071-4ef7-9ea9-a8a70b174113" (UID: "50f30b3b-6071-4ef7-9ea9-a8a70b174113"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:00:37.966298 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.966191 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-data" (OuterVolumeSpecName: "istio-data") pod "50f30b3b-6071-4ef7-9ea9-a8a70b174113" (UID: "50f30b3b-6071-4ef7-9ea9-a8a70b174113"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:00:37.967655 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.967632 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "50f30b3b-6071-4ef7-9ea9-a8a70b174113" (UID: "50f30b3b-6071-4ef7-9ea9-a8a70b174113"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:00:37.967655 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.967638 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "50f30b3b-6071-4ef7-9ea9-a8a70b174113" (UID: "50f30b3b-6071-4ef7-9ea9-a8a70b174113"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 01 01:00:37.968092 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.968069 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f30b3b-6071-4ef7-9ea9-a8a70b174113-kube-api-access-5x997" (OuterVolumeSpecName: "kube-api-access-5x997") pod "50f30b3b-6071-4ef7-9ea9-a8a70b174113" (UID: "50f30b3b-6071-4ef7-9ea9-a8a70b174113"). InnerVolumeSpecName "kube-api-access-5x997". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:00:37.968186 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:37.968074 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-token" (OuterVolumeSpecName: "istio-token") pod "50f30b3b-6071-4ef7-9ea9-a8a70b174113" (UID: "50f30b3b-6071-4ef7-9ea9-a8a70b174113"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:00:38.067121 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.067086 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-podinfo\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:38.067121 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.067116 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-credential-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:38.067121 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.067126 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-data\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:38.067353 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.067135 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-token\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:38.067353 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.067144 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-workload-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:38.067353 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.067152 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x997\" (UniqueName: \"kubernetes.io/projected/50f30b3b-6071-4ef7-9ea9-a8a70b174113-kube-api-access-5x997\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:38.067353 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.067160 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f30b3b-6071-4ef7-9ea9-a8a70b174113-istio-envoy\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:00:38.609804 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.609770 2563 generic.go:356] "Generic (PLEG): container finished" podID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerID="ec6cf9ecabd0f3cca86644e21a5a670eadbb9068963eb228dca12e1a8cbaea2b" exitCode=0 Aug 01 01:00:38.610278 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.609838 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" Aug 01 01:00:38.610278 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.609857 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" event={"ID":"50f30b3b-6071-4ef7-9ea9-a8a70b174113","Type":"ContainerDied","Data":"ec6cf9ecabd0f3cca86644e21a5a670eadbb9068963eb228dca12e1a8cbaea2b"} Aug 01 01:00:38.610278 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.609924 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts" event={"ID":"50f30b3b-6071-4ef7-9ea9-a8a70b174113","Type":"ContainerDied","Data":"f6608ef33d39b218ca2a0a2f77a7e0ed438ab72da9dcb60c9528c4e9d1987c34"} Aug 01 01:00:38.610278 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.609952 2563 scope.go:117] "RemoveContainer" containerID="ec6cf9ecabd0f3cca86644e21a5a670eadbb9068963eb228dca12e1a8cbaea2b" Aug 01 01:00:38.619359 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.619337 2563 scope.go:117] "RemoveContainer" containerID="192e77dade2949b6429c63546550296b75ae7bfd0bbd4bf9195fdd182a54b267" Aug 01 01:00:38.627585 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.627567 2563 scope.go:117] "RemoveContainer" containerID="ec6cf9ecabd0f3cca86644e21a5a670eadbb9068963eb228dca12e1a8cbaea2b" Aug 01 01:00:38.627816 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:00:38.627800 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6cf9ecabd0f3cca86644e21a5a670eadbb9068963eb228dca12e1a8cbaea2b\": container with ID starting with ec6cf9ecabd0f3cca86644e21a5a670eadbb9068963eb228dca12e1a8cbaea2b not found: ID does not exist" containerID="ec6cf9ecabd0f3cca86644e21a5a670eadbb9068963eb228dca12e1a8cbaea2b" Aug 01 01:00:38.627869 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.627824 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6cf9ecabd0f3cca86644e21a5a670eadbb9068963eb228dca12e1a8cbaea2b"} err="failed to get container status \"ec6cf9ecabd0f3cca86644e21a5a670eadbb9068963eb228dca12e1a8cbaea2b\": rpc error: code = NotFound desc = could not find container \"ec6cf9ecabd0f3cca86644e21a5a670eadbb9068963eb228dca12e1a8cbaea2b\": container with ID starting with ec6cf9ecabd0f3cca86644e21a5a670eadbb9068963eb228dca12e1a8cbaea2b not found: ID does not exist" Aug 01 01:00:38.627869 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.627840 2563 scope.go:117] "RemoveContainer" containerID="192e77dade2949b6429c63546550296b75ae7bfd0bbd4bf9195fdd182a54b267" Aug 01 01:00:38.628027 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:00:38.628011 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192e77dade2949b6429c63546550296b75ae7bfd0bbd4bf9195fdd182a54b267\": container with ID starting with 192e77dade2949b6429c63546550296b75ae7bfd0bbd4bf9195fdd182a54b267 not found: ID does not exist" containerID="192e77dade2949b6429c63546550296b75ae7bfd0bbd4bf9195fdd182a54b267" Aug 01 01:00:38.628087 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.628030 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192e77dade2949b6429c63546550296b75ae7bfd0bbd4bf9195fdd182a54b267"} err="failed to get container status \"192e77dade2949b6429c63546550296b75ae7bfd0bbd4bf9195fdd182a54b267\": rpc error: code = NotFound desc = could not find container \"192e77dade2949b6429c63546550296b75ae7bfd0bbd4bf9195fdd182a54b267\": container with ID starting with 192e77dade2949b6429c63546550296b75ae7bfd0bbd4bf9195fdd182a54b267 not found: ID does not exist" Aug 01 01:00:38.636462 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.636376 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts"] Aug 01 01:00:38.638609 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:38.638588 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c7aeac35-openshift-default-7fb974d648-mxtts"] Aug 01 01:00:39.728681 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:00:39.728645 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" path="/var/lib/kubelet/pods/50f30b3b-6071-4ef7-9ea9-a8a70b174113/volumes" Aug 01 01:02:30.200897 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.200863 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs"] Aug 01 01:02:30.201384 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201328 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2bad7cf-6886-4b31-8d89-3eceb8f24b5c" containerName="maas-api" Aug 01 01:02:30.201384 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201345 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2bad7cf-6886-4b31-8d89-3eceb8f24b5c" containerName="maas-api" Aug 01 01:02:30.201384 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201359 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7088e54f-ecb7-4743-a345-d20230e59445" containerName="istio-proxy" Aug 01 01:02:30.201384 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201365 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7088e54f-ecb7-4743-a345-d20230e59445" containerName="istio-proxy" Aug 01 01:02:30.201384 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201381 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7088e54f-ecb7-4743-a345-d20230e59445" containerName="istio-proxy" Aug 01 01:02:30.201384 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201387 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7088e54f-ecb7-4743-a345-d20230e59445" containerName="istio-proxy" Aug 01 01:02:30.201594 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201401 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" Aug 01 01:02:30.201594 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201407 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" Aug 01 01:02:30.201594 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201422 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" Aug 01 01:02:30.201594 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201431 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" Aug 01 01:02:30.201594 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201511 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7088e54f-ecb7-4743-a345-d20230e59445" containerName="istio-proxy" Aug 01 01:02:30.201594 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201524 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2bad7cf-6886-4b31-8d89-3eceb8f24b5c" containerName="maas-api" Aug 01 01:02:30.201594 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201530 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" Aug 01 01:02:30.201594 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.201538 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f30b3b-6071-4ef7-9ea9-a8a70b174113" containerName="istio-proxy" Aug 01 01:02:30.204434 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.204418 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.207483 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.207462 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b7da259d-openshift-default-dockercfg-nkqv5\"" Aug 01 01:02:30.227970 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.227945 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs"] Aug 01 01:02:30.236156 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.236130 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1758926f-9e70-468f-bfcb-2ed40e791778-istiod-ca-cert\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.236290 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.236179 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-workload-socket\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.236290 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.236233 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-istio-envoy\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.236290 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.236268 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1758926f-9e70-468f-bfcb-2ed40e791778-istio-token\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.236397 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.236326 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-workload-certs\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.236397 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.236368 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-credential-socket\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.236472 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.236405 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1758926f-9e70-468f-bfcb-2ed40e791778-istio-podinfo\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.236472 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.236425 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-istio-data\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.236549 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.236483 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg6m9\" (UniqueName: \"kubernetes.io/projected/1758926f-9e70-468f-bfcb-2ed40e791778-kube-api-access-gg6m9\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.337368 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.337332 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-workload-certs\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.337368 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.337376 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-credential-socket\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.337605 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.337402 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1758926f-9e70-468f-bfcb-2ed40e791778-istio-podinfo\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.337605 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.337426 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-istio-data\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.337605 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.337455 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg6m9\" (UniqueName: \"kubernetes.io/projected/1758926f-9e70-468f-bfcb-2ed40e791778-kube-api-access-gg6m9\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.337605 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.337493 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1758926f-9e70-468f-bfcb-2ed40e791778-istiod-ca-cert\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.337605 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.337535 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-workload-socket\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.337605 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.337561 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-istio-envoy\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.337605 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.337590 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1758926f-9e70-468f-bfcb-2ed40e791778-istio-token\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.338008 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.337822 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-workload-certs\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.338008 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.337855 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-credential-socket\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.338008 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.337935 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-workload-socket\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.338190 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.338160 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-istio-data\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.338324 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.338302 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1758926f-9e70-468f-bfcb-2ed40e791778-istiod-ca-cert\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.339752 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.339727 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-istio-envoy\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.339868 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.339851 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1758926f-9e70-468f-bfcb-2ed40e791778-istio-podinfo\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.345738 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.345713 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1758926f-9e70-468f-bfcb-2ed40e791778-istio-token\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.345840 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.345818 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg6m9\" (UniqueName: \"kubernetes.io/projected/1758926f-9e70-468f-bfcb-2ed40e791778-kube-api-access-gg6m9\") pod \"e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.522297 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.522212 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:30.662226 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.662195 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs"] Aug 01 01:02:30.664119 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:02:30.664091 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1758926f_9e70_468f_bfcb_2ed40e791778.slice/crio-aebfa718b1fa921e13405e121e997f66323a4722c068bde261cc29bc10f406ed WatchSource:0}: Error finding container aebfa718b1fa921e13405e121e997f66323a4722c068bde261cc29bc10f406ed: Status 404 returned error can't find the container with id aebfa718b1fa921e13405e121e997f66323a4722c068bde261cc29bc10f406ed Aug 01 01:02:30.666377 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.666342 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:02:30.666475 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.666424 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:02:30.666475 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:30.666466 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:02:31.142367 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:31.142331 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" event={"ID":"1758926f-9e70-468f-bfcb-2ed40e791778","Type":"ContainerStarted","Data":"8c844cc82805fa33a2ada5a33f7179fc27dd0b74124a182e35a76db1201c0427"} Aug 01 01:02:31.142543 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:31.142397 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" event={"ID":"1758926f-9e70-468f-bfcb-2ed40e791778","Type":"ContainerStarted","Data":"aebfa718b1fa921e13405e121e997f66323a4722c068bde261cc29bc10f406ed"} Aug 01 01:02:31.164832 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:31.164777 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" podStartSLOduration=1.164760675 podStartE2EDuration="1.164760675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:02:31.163184855 +0000 UTC m=+2039.951906693" watchObservedRunningTime="2026-08-01 01:02:31.164760675 +0000 UTC m=+2039.953482501" Aug 01 01:02:31.523288 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:31.523190 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:31.527958 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:31.527931 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:32.148068 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:32.148035 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:32.149081 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:32.149038 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:34.170122 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.170043 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98"] Aug 01 01:02:34.170705 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.170687 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7088e54f-ecb7-4743-a345-d20230e59445" containerName="istio-proxy" Aug 01 01:02:34.174365 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.174346 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:02:34.177238 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.177210 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b7da259d-serving-cert\"" Aug 01 01:02:34.178529 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.178491 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 01 01:02:34.178529 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.178512 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:02:34.178713 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.178512 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gf2hx\"" Aug 01 01:02:34.190717 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.190694 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98"] Aug 01 01:02:34.275976 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.275935 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95556b5b-c948-4f44-8d6e-943934066a59-openshift-service-ca\") pod \"maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98\" (UID: \"95556b5b-c948-4f44-8d6e-943934066a59\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:02:34.276177 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.276003 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbv4n\" (UniqueName: \"kubernetes.io/projected/95556b5b-c948-4f44-8d6e-943934066a59-kube-api-access-nbv4n\") pod \"maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98\" (UID: \"95556b5b-c948-4f44-8d6e-943934066a59\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:02:34.276177 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.276029 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95556b5b-c948-4f44-8d6e-943934066a59-maas-api-tls\") pod \"maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98\" (UID: \"95556b5b-c948-4f44-8d6e-943934066a59\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:02:34.377380 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.377337 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95556b5b-c948-4f44-8d6e-943934066a59-openshift-service-ca\") pod \"maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98\" (UID: \"95556b5b-c948-4f44-8d6e-943934066a59\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:02:34.377555 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.377392 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbv4n\" (UniqueName: \"kubernetes.io/projected/95556b5b-c948-4f44-8d6e-943934066a59-kube-api-access-nbv4n\") pod \"maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98\" (UID: \"95556b5b-c948-4f44-8d6e-943934066a59\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:02:34.377555 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.377422 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95556b5b-c948-4f44-8d6e-943934066a59-maas-api-tls\") pod \"maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98\" (UID: \"95556b5b-c948-4f44-8d6e-943934066a59\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:02:34.378088 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.378037 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95556b5b-c948-4f44-8d6e-943934066a59-openshift-service-ca\") pod \"maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98\" (UID: \"95556b5b-c948-4f44-8d6e-943934066a59\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:02:34.379896 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.379865 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95556b5b-c948-4f44-8d6e-943934066a59-maas-api-tls\") pod \"maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98\" (UID: \"95556b5b-c948-4f44-8d6e-943934066a59\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:02:34.391919 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.391891 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbv4n\" (UniqueName: \"kubernetes.io/projected/95556b5b-c948-4f44-8d6e-943934066a59-kube-api-access-nbv4n\") pod \"maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98\" (UID: \"95556b5b-c948-4f44-8d6e-943934066a59\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:02:34.485129 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.485016 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:02:34.631446 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.631421 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98"] Aug 01 01:02:34.633520 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:02:34.633486 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95556b5b_c948_4f44_8d6e_943934066a59.slice/crio-523066b4b848773018917600126562991b2e68416ed632465600824bae847c3f WatchSource:0}: Error finding container 523066b4b848773018917600126562991b2e68416ed632465600824bae847c3f: Status 404 returned error can't find the container with id 523066b4b848773018917600126562991b2e68416ed632465600824bae847c3f Aug 01 01:02:34.964645 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.964607 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5"] Aug 01 01:02:34.969496 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.969463 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" Aug 01 01:02:34.973453 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.973430 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b7da259d-dockercfg-bnj5g\"" Aug 01 01:02:34.973588 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.973471 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b7da259d\"" Aug 01 01:02:34.987155 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:34.987126 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5"] Aug 01 01:02:35.018285 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.018254 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz"] Aug 01 01:02:35.022126 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.022101 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" Aug 01 01:02:35.036214 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.036192 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz"] Aug 01 01:02:35.085314 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.085282 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfb2\" (UniqueName: \"kubernetes.io/projected/72d880e7-99e4-432e-9f94-be27217d7aa6-kube-api-access-rvfb2\") pod \"payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz\" (UID: \"72d880e7-99e4-432e-9f94-be27217d7aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" Aug 01 01:02:35.085485 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.085348 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96fdba51-bf80-424b-8d22-907eac7a700a-plugins-config-volume\") pod \"payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5\" (UID: \"96fdba51-bf80-424b-8d22-907eac7a700a\") " pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" Aug 01 01:02:35.085485 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.085393 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d880e7-99e4-432e-9f94-be27217d7aa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz\" (UID: \"72d880e7-99e4-432e-9f94-be27217d7aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" Aug 01 01:02:35.085485 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.085457 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pksk\" (UniqueName: \"kubernetes.io/projected/96fdba51-bf80-424b-8d22-907eac7a700a-kube-api-access-2pksk\") pod \"payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5\" (UID: \"96fdba51-bf80-424b-8d22-907eac7a700a\") " pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" Aug 01 01:02:35.165542 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.165511 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" event={"ID":"95556b5b-c948-4f44-8d6e-943934066a59","Type":"ContainerStarted","Data":"523066b4b848773018917600126562991b2e68416ed632465600824bae847c3f"} Aug 01 01:02:35.186371 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.186338 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d880e7-99e4-432e-9f94-be27217d7aa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz\" (UID: \"72d880e7-99e4-432e-9f94-be27217d7aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" Aug 01 01:02:35.186690 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.186381 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pksk\" (UniqueName: \"kubernetes.io/projected/96fdba51-bf80-424b-8d22-907eac7a700a-kube-api-access-2pksk\") pod \"payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5\" (UID: \"96fdba51-bf80-424b-8d22-907eac7a700a\") " pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" Aug 01 01:02:35.186690 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.186445 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfb2\" (UniqueName: \"kubernetes.io/projected/72d880e7-99e4-432e-9f94-be27217d7aa6-kube-api-access-rvfb2\") pod \"payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz\" (UID: \"72d880e7-99e4-432e-9f94-be27217d7aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" Aug 01 01:02:35.186690 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.186489 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96fdba51-bf80-424b-8d22-907eac7a700a-plugins-config-volume\") pod \"payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5\" (UID: \"96fdba51-bf80-424b-8d22-907eac7a700a\") " pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" Aug 01 01:02:35.187190 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.187174 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96fdba51-bf80-424b-8d22-907eac7a700a-plugins-config-volume\") pod \"payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5\" (UID: \"96fdba51-bf80-424b-8d22-907eac7a700a\") " pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" Aug 01 01:02:35.187236 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.187206 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d880e7-99e4-432e-9f94-be27217d7aa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz\" (UID: \"72d880e7-99e4-432e-9f94-be27217d7aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" Aug 01 01:02:35.195088 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.195048 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pksk\" (UniqueName: \"kubernetes.io/projected/96fdba51-bf80-424b-8d22-907eac7a700a-kube-api-access-2pksk\") pod \"payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5\" (UID: \"96fdba51-bf80-424b-8d22-907eac7a700a\") " pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" Aug 01 01:02:35.195174 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.195109 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfb2\" (UniqueName: \"kubernetes.io/projected/72d880e7-99e4-432e-9f94-be27217d7aa6-kube-api-access-rvfb2\") pod \"payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz\" (UID: \"72d880e7-99e4-432e-9f94-be27217d7aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" Aug 01 01:02:35.279957 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.279868 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" Aug 01 01:02:35.332515 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.332436 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" Aug 01 01:02:35.427932 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.427906 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5"] Aug 01 01:02:35.430569 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:02:35.430515 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96fdba51_bf80_424b_8d22_907eac7a700a.slice/crio-57f5680eed2db8a1601f977a73b9c75edf3b7ce5b5fb2c84f77a2dfcbf35f659 WatchSource:0}: Error finding container 57f5680eed2db8a1601f977a73b9c75edf3b7ce5b5fb2c84f77a2dfcbf35f659: Status 404 returned error can't find the container with id 57f5680eed2db8a1601f977a73b9c75edf3b7ce5b5fb2c84f77a2dfcbf35f659 Aug 01 01:02:35.490172 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:35.490146 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz"] Aug 01 01:02:35.491635 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:02:35.491610 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d880e7_99e4_432e_9f94_be27217d7aa6.slice/crio-3a0f4505f9a8386f541256b91d4bc3e9213a072b70d32fc433cbfdd8cd903b06 WatchSource:0}: Error finding container 3a0f4505f9a8386f541256b91d4bc3e9213a072b70d32fc433cbfdd8cd903b06: Status 404 returned error can't find the container with id 3a0f4505f9a8386f541256b91d4bc3e9213a072b70d32fc433cbfdd8cd903b06 Aug 01 01:02:36.171770 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:36.171730 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" event={"ID":"96fdba51-bf80-424b-8d22-907eac7a700a","Type":"ContainerStarted","Data":"eb80729c5c4013f069caaaea94a119d127fdf3957332ab8345396d5f8da5ce00"} Aug 01 01:02:36.171991 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:36.171797 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" event={"ID":"96fdba51-bf80-424b-8d22-907eac7a700a","Type":"ContainerStarted","Data":"57f5680eed2db8a1601f977a73b9c75edf3b7ce5b5fb2c84f77a2dfcbf35f659"} Aug 01 01:02:36.172082 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:36.171993 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" Aug 01 01:02:36.176039 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:36.176010 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" event={"ID":"95556b5b-c948-4f44-8d6e-943934066a59","Type":"ContainerStarted","Data":"460fa5d6c3af961767cc294782dc9470d37eb129d8ba782658753ffc5f7e3268"} Aug 01 01:02:36.176193 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:36.176079 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:02:36.177550 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:36.177527 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" event={"ID":"72d880e7-99e4-432e-9f94-be27217d7aa6","Type":"ContainerStarted","Data":"ee306f9c8e9a0d233c819a52810e365d6601a0d8e771f6189529b15bc16ed657"} Aug 01 01:02:36.177653 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:36.177571 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" event={"ID":"72d880e7-99e4-432e-9f94-be27217d7aa6","Type":"ContainerStarted","Data":"3a0f4505f9a8386f541256b91d4bc3e9213a072b70d32fc433cbfdd8cd903b06"} Aug 01 01:02:36.177653 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:36.177594 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" Aug 01 01:02:36.202696 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:36.202653 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" podStartSLOduration=2.202639562 podStartE2EDuration="2.202639562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:02:36.201400464 +0000 UTC m=+2044.990122289" watchObservedRunningTime="2026-08-01 01:02:36.202639562 +0000 UTC m=+2044.991361388" Aug 01 01:02:36.235122 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:36.235038 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" podStartSLOduration=1.781223395 podStartE2EDuration="2.235024098s" totalImagesPullingTime="453.800703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:02:36.234847196 +0000 UTC m=+2045.023569019" watchObservedRunningTime="2026-08-01 01:02:36.235024098 +0000 UTC m=+2045.023745918" Aug 01 01:02:36.264585 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:36.264539 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" podStartSLOduration=2.264524819 podStartE2EDuration="2.264524819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:02:36.262877693 +0000 UTC m=+2045.051599518" watchObservedRunningTime="2026-08-01 01:02:36.264524819 +0000 UTC m=+2045.053246647" Aug 01 01:02:42.187427 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:42.187400 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:02:47.150151 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:47.150115 2563 patch_prober.go:28] interesting pod/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 01 01:02:47.150557 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:47.150167 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 01 01:02:47.184802 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:47.184779 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" Aug 01 01:02:47.184968 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:47.184844 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" Aug 01 01:02:47.234301 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:47.234267 2563 generic.go:356] "Generic (PLEG): container finished" podID="1758926f-9e70-468f-bfcb-2ed40e791778" containerID="8c844cc82805fa33a2ada5a33f7179fc27dd0b74124a182e35a76db1201c0427" exitCode=0 Aug 01 01:02:47.234301 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:47.234313 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" event={"ID":"1758926f-9e70-468f-bfcb-2ed40e791778","Type":"ContainerDied","Data":"8c844cc82805fa33a2ada5a33f7179fc27dd0b74124a182e35a76db1201c0427"} Aug 01 01:02:47.234848 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:47.234818 2563 scope.go:117] "RemoveContainer" containerID="8c844cc82805fa33a2ada5a33f7179fc27dd0b74124a182e35a76db1201c0427" Aug 01 01:02:47.235907 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:47.235878 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:02:47.235969 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:47.235955 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:02:47.236022 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:47.236008 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:02:48.242397 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:48.242358 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" event={"ID":"1758926f-9e70-468f-bfcb-2ed40e791778","Type":"ContainerStarted","Data":"307a48a1e501b12ece18b53202cc5be2f2ddbe4f5cc626dba7c1665f42250084"} Aug 01 01:02:48.523156 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:48.523036 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:48.523553 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:48.523429 2563 patch_prober.go:28] interesting pod/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 01 01:02:48.523770 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:48.523745 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 01 01:02:49.523565 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:49.523519 2563 patch_prober.go:28] interesting pod/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 01 01:02:49.524086 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:49.523607 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 01 01:02:50.523489 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:50.523445 2563 patch_prober.go:28] interesting pod/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 01 01:02:50.523685 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:50.523516 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 01 01:02:51.523435 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:51.523397 2563 patch_prober.go:28] interesting pod/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 01 01:02:51.523736 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:51.523705 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 01 01:02:52.523566 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:52.523517 2563 patch_prober.go:28] interesting pod/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 01 01:02:52.523759 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:52.523591 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 01 01:02:53.527172 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:53.527135 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:53.527595 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:53.527399 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:02:53.528146 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:02:53.528127 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:03:15.996143 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:15.996046 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98"] Aug 01 01:03:15.996681 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:15.996292 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" podUID="95556b5b-c948-4f44-8d6e-943934066a59" containerName="maas-api" containerID="cri-o://460fa5d6c3af961767cc294782dc9470d37eb129d8ba782658753ffc5f7e3268" gracePeriod=30 Aug 01 01:03:16.074955 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.074871 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5"] Aug 01 01:03:16.075289 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.075239 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" podUID="96fdba51-bf80-424b-8d22-907eac7a700a" containerName="payload-processing" containerID="cri-o://eb80729c5c4013f069caaaea94a119d127fdf3957332ab8345396d5f8da5ce00" gracePeriod=30 Aug 01 01:03:16.082028 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.082005 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz"] Aug 01 01:03:16.082259 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.082237 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" podUID="72d880e7-99e4-432e-9f94-be27217d7aa6" containerName="payload-pre-processing" containerID="cri-o://ee306f9c8e9a0d233c819a52810e365d6601a0d8e771f6189529b15bc16ed657" gracePeriod=30 Aug 01 01:03:16.353730 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.353702 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:03:16.383521 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.383490 2563 generic.go:356] "Generic (PLEG): container finished" podID="95556b5b-c948-4f44-8d6e-943934066a59" containerID="460fa5d6c3af961767cc294782dc9470d37eb129d8ba782658753ffc5f7e3268" exitCode=0 Aug 01 01:03:16.383667 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.383537 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" Aug 01 01:03:16.383667 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.383582 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" event={"ID":"95556b5b-c948-4f44-8d6e-943934066a59","Type":"ContainerDied","Data":"460fa5d6c3af961767cc294782dc9470d37eb129d8ba782658753ffc5f7e3268"} Aug 01 01:03:16.383667 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.383640 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98" event={"ID":"95556b5b-c948-4f44-8d6e-943934066a59","Type":"ContainerDied","Data":"523066b4b848773018917600126562991b2e68416ed632465600824bae847c3f"} Aug 01 01:03:16.383828 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.383669 2563 scope.go:117] "RemoveContainer" containerID="460fa5d6c3af961767cc294782dc9470d37eb129d8ba782658753ffc5f7e3268" Aug 01 01:03:16.384025 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.383955 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" Aug 01 01:03:16.385129 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.385109 2563 generic.go:356] "Generic (PLEG): container finished" podID="72d880e7-99e4-432e-9f94-be27217d7aa6" containerID="ee306f9c8e9a0d233c819a52810e365d6601a0d8e771f6189529b15bc16ed657" exitCode=0 Aug 01 01:03:16.385239 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.385186 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" event={"ID":"72d880e7-99e4-432e-9f94-be27217d7aa6","Type":"ContainerDied","Data":"ee306f9c8e9a0d233c819a52810e365d6601a0d8e771f6189529b15bc16ed657"} Aug 01 01:03:16.386701 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.386679 2563 generic.go:356] "Generic (PLEG): container finished" podID="96fdba51-bf80-424b-8d22-907eac7a700a" containerID="eb80729c5c4013f069caaaea94a119d127fdf3957332ab8345396d5f8da5ce00" exitCode=0 Aug 01 01:03:16.386801 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.386714 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" event={"ID":"96fdba51-bf80-424b-8d22-907eac7a700a","Type":"ContainerDied","Data":"eb80729c5c4013f069caaaea94a119d127fdf3957332ab8345396d5f8da5ce00"} Aug 01 01:03:16.386801 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.386749 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" event={"ID":"96fdba51-bf80-424b-8d22-907eac7a700a","Type":"ContainerDied","Data":"57f5680eed2db8a1601f977a73b9c75edf3b7ce5b5fb2c84f77a2dfcbf35f659"} Aug 01 01:03:16.386903 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.386806 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5" Aug 01 01:03:16.389391 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.389372 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" Aug 01 01:03:16.394429 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.394277 2563 scope.go:117] "RemoveContainer" containerID="460fa5d6c3af961767cc294782dc9470d37eb129d8ba782658753ffc5f7e3268" Aug 01 01:03:16.396417 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:03:16.394849 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460fa5d6c3af961767cc294782dc9470d37eb129d8ba782658753ffc5f7e3268\": container with ID starting with 460fa5d6c3af961767cc294782dc9470d37eb129d8ba782658753ffc5f7e3268 not found: ID does not exist" containerID="460fa5d6c3af961767cc294782dc9470d37eb129d8ba782658753ffc5f7e3268" Aug 01 01:03:16.396417 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.394893 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460fa5d6c3af961767cc294782dc9470d37eb129d8ba782658753ffc5f7e3268"} err="failed to get container status \"460fa5d6c3af961767cc294782dc9470d37eb129d8ba782658753ffc5f7e3268\": rpc error: code = NotFound desc = could not find container \"460fa5d6c3af961767cc294782dc9470d37eb129d8ba782658753ffc5f7e3268\": container with ID starting with 460fa5d6c3af961767cc294782dc9470d37eb129d8ba782658753ffc5f7e3268 not found: ID does not exist" Aug 01 01:03:16.396417 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.394917 2563 scope.go:117] "RemoveContainer" containerID="eb80729c5c4013f069caaaea94a119d127fdf3957332ab8345396d5f8da5ce00" Aug 01 01:03:16.407249 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.407226 2563 scope.go:117] "RemoveContainer" containerID="eb80729c5c4013f069caaaea94a119d127fdf3957332ab8345396d5f8da5ce00" Aug 01 01:03:16.407591 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:03:16.407557 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb80729c5c4013f069caaaea94a119d127fdf3957332ab8345396d5f8da5ce00\": container with ID starting with eb80729c5c4013f069caaaea94a119d127fdf3957332ab8345396d5f8da5ce00 not found: ID does not exist" containerID="eb80729c5c4013f069caaaea94a119d127fdf3957332ab8345396d5f8da5ce00" Aug 01 01:03:16.407708 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.407598 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb80729c5c4013f069caaaea94a119d127fdf3957332ab8345396d5f8da5ce00"} err="failed to get container status \"eb80729c5c4013f069caaaea94a119d127fdf3957332ab8345396d5f8da5ce00\": rpc error: code = NotFound desc = could not find container \"eb80729c5c4013f069caaaea94a119d127fdf3957332ab8345396d5f8da5ce00\": container with ID starting with eb80729c5c4013f069caaaea94a119d127fdf3957332ab8345396d5f8da5ce00 not found: ID does not exist" Aug 01 01:03:16.476596 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.476565 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95556b5b-c948-4f44-8d6e-943934066a59-maas-api-tls\") pod \"95556b5b-c948-4f44-8d6e-943934066a59\" (UID: \"95556b5b-c948-4f44-8d6e-943934066a59\") " Aug 01 01:03:16.476754 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.476664 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d880e7-99e4-432e-9f94-be27217d7aa6-plugins-config-volume\") pod \"72d880e7-99e4-432e-9f94-be27217d7aa6\" (UID: \"72d880e7-99e4-432e-9f94-be27217d7aa6\") " Aug 01 01:03:16.476754 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.476711 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96fdba51-bf80-424b-8d22-907eac7a700a-plugins-config-volume\") pod \"96fdba51-bf80-424b-8d22-907eac7a700a\" (UID: \"96fdba51-bf80-424b-8d22-907eac7a700a\") " Aug 01 01:03:16.476754 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.476730 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95556b5b-c948-4f44-8d6e-943934066a59-openshift-service-ca\") pod \"95556b5b-c948-4f44-8d6e-943934066a59\" (UID: \"95556b5b-c948-4f44-8d6e-943934066a59\") " Aug 01 01:03:16.476894 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.476762 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pksk\" (UniqueName: \"kubernetes.io/projected/96fdba51-bf80-424b-8d22-907eac7a700a-kube-api-access-2pksk\") pod \"96fdba51-bf80-424b-8d22-907eac7a700a\" (UID: \"96fdba51-bf80-424b-8d22-907eac7a700a\") " Aug 01 01:03:16.476894 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.476792 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbv4n\" (UniqueName: \"kubernetes.io/projected/95556b5b-c948-4f44-8d6e-943934066a59-kube-api-access-nbv4n\") pod \"95556b5b-c948-4f44-8d6e-943934066a59\" (UID: \"95556b5b-c948-4f44-8d6e-943934066a59\") " Aug 01 01:03:16.476894 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.476820 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvfb2\" (UniqueName: \"kubernetes.io/projected/72d880e7-99e4-432e-9f94-be27217d7aa6-kube-api-access-rvfb2\") pod \"72d880e7-99e4-432e-9f94-be27217d7aa6\" (UID: \"72d880e7-99e4-432e-9f94-be27217d7aa6\") " Aug 01 01:03:16.477089 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.477040 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d880e7-99e4-432e-9f94-be27217d7aa6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72d880e7-99e4-432e-9f94-be27217d7aa6" (UID: "72d880e7-99e4-432e-9f94-be27217d7aa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:03:16.477178 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.477114 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96fdba51-bf80-424b-8d22-907eac7a700a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96fdba51-bf80-424b-8d22-907eac7a700a" (UID: "96fdba51-bf80-424b-8d22-907eac7a700a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:03:16.477178 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.477138 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95556b5b-c948-4f44-8d6e-943934066a59-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95556b5b-c948-4f44-8d6e-943934066a59" (UID: "95556b5b-c948-4f44-8d6e-943934066a59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:03:16.477290 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.477198 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d880e7-99e4-432e-9f94-be27217d7aa6-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:16.478835 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.478811 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fdba51-bf80-424b-8d22-907eac7a700a-kube-api-access-2pksk" (OuterVolumeSpecName: "kube-api-access-2pksk") pod "96fdba51-bf80-424b-8d22-907eac7a700a" (UID: "96fdba51-bf80-424b-8d22-907eac7a700a"). InnerVolumeSpecName "kube-api-access-2pksk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:03:16.478945 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.478858 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95556b5b-c948-4f44-8d6e-943934066a59-kube-api-access-nbv4n" (OuterVolumeSpecName: "kube-api-access-nbv4n") pod "95556b5b-c948-4f44-8d6e-943934066a59" (UID: "95556b5b-c948-4f44-8d6e-943934066a59"). InnerVolumeSpecName "kube-api-access-nbv4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:03:16.479071 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.479045 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95556b5b-c948-4f44-8d6e-943934066a59-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95556b5b-c948-4f44-8d6e-943934066a59" (UID: "95556b5b-c948-4f44-8d6e-943934066a59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:03:16.479181 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.479165 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d880e7-99e4-432e-9f94-be27217d7aa6-kube-api-access-rvfb2" (OuterVolumeSpecName: "kube-api-access-rvfb2") pod "72d880e7-99e4-432e-9f94-be27217d7aa6" (UID: "72d880e7-99e4-432e-9f94-be27217d7aa6"). InnerVolumeSpecName "kube-api-access-rvfb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:03:16.578403 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.578373 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96fdba51-bf80-424b-8d22-907eac7a700a-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:16.578403 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.578400 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95556b5b-c948-4f44-8d6e-943934066a59-openshift-service-ca\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:16.578403 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.578409 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pksk\" (UniqueName: \"kubernetes.io/projected/96fdba51-bf80-424b-8d22-907eac7a700a-kube-api-access-2pksk\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:16.578615 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.578418 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbv4n\" (UniqueName: \"kubernetes.io/projected/95556b5b-c948-4f44-8d6e-943934066a59-kube-api-access-nbv4n\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:16.578615 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.578427 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvfb2\" (UniqueName: \"kubernetes.io/projected/72d880e7-99e4-432e-9f94-be27217d7aa6-kube-api-access-rvfb2\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:16.578615 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.578435 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95556b5b-c948-4f44-8d6e-943934066a59-maas-api-tls\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:16.707323 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.707297 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98"] Aug 01 01:03:16.715230 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.715203 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7da259d-6d9c64c79c-24w98"] Aug 01 01:03:16.725922 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.725899 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5"] Aug 01 01:03:16.732485 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:16.732466 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b7da259d-75dd8f5694-ddnt5"] Aug 01 01:03:17.393335 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:17.393304 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" Aug 01 01:03:17.393784 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:17.393350 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz" event={"ID":"72d880e7-99e4-432e-9f94-be27217d7aa6","Type":"ContainerDied","Data":"3a0f4505f9a8386f541256b91d4bc3e9213a072b70d32fc433cbfdd8cd903b06"} Aug 01 01:03:17.393784 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:17.393413 2563 scope.go:117] "RemoveContainer" containerID="ee306f9c8e9a0d233c819a52810e365d6601a0d8e771f6189529b15bc16ed657" Aug 01 01:03:17.419153 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:17.419122 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz"] Aug 01 01:03:17.422904 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:17.422870 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b7da259d-76cbf7bc5b-pg5hz"] Aug 01 01:03:17.728504 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:17.728418 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72d880e7-99e4-432e-9f94-be27217d7aa6" path="/var/lib/kubelet/pods/72d880e7-99e4-432e-9f94-be27217d7aa6/volumes" Aug 01 01:03:17.728817 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:17.728804 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95556b5b-c948-4f44-8d6e-943934066a59" path="/var/lib/kubelet/pods/95556b5b-c948-4f44-8d6e-943934066a59/volumes" Aug 01 01:03:17.729134 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:17.729120 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96fdba51-bf80-424b-8d22-907eac7a700a" path="/var/lib/kubelet/pods/96fdba51-bf80-424b-8d22-907eac7a700a/volumes" Aug 01 01:03:23.419103 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:23.419068 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs"] Aug 01 01:03:23.419569 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:23.419361 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" containerID="cri-o://307a48a1e501b12ece18b53202cc5be2f2ddbe4f5cc626dba7c1665f42250084" gracePeriod=30 Aug 01 01:03:23.530197 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:23.530161 2563 patch_prober.go:28] interesting pod/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 01 01:03:23.530377 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:23.530249 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 01 01:03:28.666808 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.666781 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:03:28.794630 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.794548 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1758926f-9e70-468f-bfcb-2ed40e791778-istio-token\") pod \"1758926f-9e70-468f-bfcb-2ed40e791778\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " Aug 01 01:03:28.794630 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.794582 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-credential-socket\") pod \"1758926f-9e70-468f-bfcb-2ed40e791778\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " Aug 01 01:03:28.794630 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.794628 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-istio-data\") pod \"1758926f-9e70-468f-bfcb-2ed40e791778\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " Aug 01 01:03:28.794897 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.794656 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-istio-envoy\") pod \"1758926f-9e70-468f-bfcb-2ed40e791778\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " Aug 01 01:03:28.794897 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.794691 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1758926f-9e70-468f-bfcb-2ed40e791778-istiod-ca-cert\") pod \"1758926f-9e70-468f-bfcb-2ed40e791778\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " Aug 01 01:03:28.794897 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.794732 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-workload-certs\") pod \"1758926f-9e70-468f-bfcb-2ed40e791778\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " Aug 01 01:03:28.794897 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.794761 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1758926f-9e70-468f-bfcb-2ed40e791778-istio-podinfo\") pod \"1758926f-9e70-468f-bfcb-2ed40e791778\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " Aug 01 01:03:28.795136 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.794896 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1758926f-9e70-468f-bfcb-2ed40e791778" (UID: "1758926f-9e70-468f-bfcb-2ed40e791778"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:03:28.795136 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.794976 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-workload-socket\") pod \"1758926f-9e70-468f-bfcb-2ed40e791778\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " Aug 01 01:03:28.795136 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.795084 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-istio-data" (OuterVolumeSpecName: "istio-data") pod "1758926f-9e70-468f-bfcb-2ed40e791778" (UID: "1758926f-9e70-468f-bfcb-2ed40e791778"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:03:28.795136 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.795108 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1758926f-9e70-468f-bfcb-2ed40e791778" (UID: "1758926f-9e70-468f-bfcb-2ed40e791778"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:03:28.795136 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.795126 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg6m9\" (UniqueName: \"kubernetes.io/projected/1758926f-9e70-468f-bfcb-2ed40e791778-kube-api-access-gg6m9\") pod \"1758926f-9e70-468f-bfcb-2ed40e791778\" (UID: \"1758926f-9e70-468f-bfcb-2ed40e791778\") " Aug 01 01:03:28.795387 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.795214 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1758926f-9e70-468f-bfcb-2ed40e791778-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1758926f-9e70-468f-bfcb-2ed40e791778" (UID: "1758926f-9e70-468f-bfcb-2ed40e791778"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:03:28.795387 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.795310 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1758926f-9e70-468f-bfcb-2ed40e791778" (UID: "1758926f-9e70-468f-bfcb-2ed40e791778"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:03:28.795506 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.795480 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1758926f-9e70-468f-bfcb-2ed40e791778-istiod-ca-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:28.795506 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.795500 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-workload-certs\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:28.795609 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.795513 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-workload-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:28.795609 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.795526 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-credential-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:28.795609 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.795539 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-istio-data\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:28.796870 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.796835 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1758926f-9e70-468f-bfcb-2ed40e791778" (UID: "1758926f-9e70-468f-bfcb-2ed40e791778"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:03:28.796985 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.796893 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1758926f-9e70-468f-bfcb-2ed40e791778-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1758926f-9e70-468f-bfcb-2ed40e791778" (UID: "1758926f-9e70-468f-bfcb-2ed40e791778"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 01 01:03:28.797028 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.796981 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1758926f-9e70-468f-bfcb-2ed40e791778-istio-token" (OuterVolumeSpecName: "istio-token") pod "1758926f-9e70-468f-bfcb-2ed40e791778" (UID: "1758926f-9e70-468f-bfcb-2ed40e791778"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:03:28.797224 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.797205 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1758926f-9e70-468f-bfcb-2ed40e791778-kube-api-access-gg6m9" (OuterVolumeSpecName: "kube-api-access-gg6m9") pod "1758926f-9e70-468f-bfcb-2ed40e791778" (UID: "1758926f-9e70-468f-bfcb-2ed40e791778"). InnerVolumeSpecName "kube-api-access-gg6m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:03:28.897013 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.896981 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg6m9\" (UniqueName: \"kubernetes.io/projected/1758926f-9e70-468f-bfcb-2ed40e791778-kube-api-access-gg6m9\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:28.897013 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.897006 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1758926f-9e70-468f-bfcb-2ed40e791778-istio-token\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:28.897013 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.897015 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1758926f-9e70-468f-bfcb-2ed40e791778-istio-envoy\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:28.897269 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:28.897026 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1758926f-9e70-468f-bfcb-2ed40e791778-istio-podinfo\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:03:29.462516 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:29.462479 2563 generic.go:356] "Generic (PLEG): container finished" podID="1758926f-9e70-468f-bfcb-2ed40e791778" containerID="307a48a1e501b12ece18b53202cc5be2f2ddbe4f5cc626dba7c1665f42250084" exitCode=0 Aug 01 01:03:29.462690 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:29.462526 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" Aug 01 01:03:29.462690 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:29.462541 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" event={"ID":"1758926f-9e70-468f-bfcb-2ed40e791778","Type":"ContainerDied","Data":"307a48a1e501b12ece18b53202cc5be2f2ddbe4f5cc626dba7c1665f42250084"} Aug 01 01:03:29.462690 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:29.462591 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs" event={"ID":"1758926f-9e70-468f-bfcb-2ed40e791778","Type":"ContainerDied","Data":"aebfa718b1fa921e13405e121e997f66323a4722c068bde261cc29bc10f406ed"} Aug 01 01:03:29.462690 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:29.462618 2563 scope.go:117] "RemoveContainer" containerID="307a48a1e501b12ece18b53202cc5be2f2ddbe4f5cc626dba7c1665f42250084" Aug 01 01:03:29.471471 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:29.471454 2563 scope.go:117] "RemoveContainer" containerID="8c844cc82805fa33a2ada5a33f7179fc27dd0b74124a182e35a76db1201c0427" Aug 01 01:03:29.479263 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:29.479241 2563 scope.go:117] "RemoveContainer" containerID="307a48a1e501b12ece18b53202cc5be2f2ddbe4f5cc626dba7c1665f42250084" Aug 01 01:03:29.479495 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:03:29.479472 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"307a48a1e501b12ece18b53202cc5be2f2ddbe4f5cc626dba7c1665f42250084\": container with ID starting with 307a48a1e501b12ece18b53202cc5be2f2ddbe4f5cc626dba7c1665f42250084 not found: ID does not exist" containerID="307a48a1e501b12ece18b53202cc5be2f2ddbe4f5cc626dba7c1665f42250084" Aug 01 01:03:29.479584 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:29.479502 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"307a48a1e501b12ece18b53202cc5be2f2ddbe4f5cc626dba7c1665f42250084"} err="failed to get container status \"307a48a1e501b12ece18b53202cc5be2f2ddbe4f5cc626dba7c1665f42250084\": rpc error: code = NotFound desc = could not find container \"307a48a1e501b12ece18b53202cc5be2f2ddbe4f5cc626dba7c1665f42250084\": container with ID starting with 307a48a1e501b12ece18b53202cc5be2f2ddbe4f5cc626dba7c1665f42250084 not found: ID does not exist" Aug 01 01:03:29.479584 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:29.479520 2563 scope.go:117] "RemoveContainer" containerID="8c844cc82805fa33a2ada5a33f7179fc27dd0b74124a182e35a76db1201c0427" Aug 01 01:03:29.479730 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:03:29.479714 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c844cc82805fa33a2ada5a33f7179fc27dd0b74124a182e35a76db1201c0427\": container with ID starting with 8c844cc82805fa33a2ada5a33f7179fc27dd0b74124a182e35a76db1201c0427 not found: ID does not exist" containerID="8c844cc82805fa33a2ada5a33f7179fc27dd0b74124a182e35a76db1201c0427" Aug 01 01:03:29.479793 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:29.479732 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c844cc82805fa33a2ada5a33f7179fc27dd0b74124a182e35a76db1201c0427"} err="failed to get container status \"8c844cc82805fa33a2ada5a33f7179fc27dd0b74124a182e35a76db1201c0427\": rpc error: code = NotFound desc = could not find container \"8c844cc82805fa33a2ada5a33f7179fc27dd0b74124a182e35a76db1201c0427\": container with ID starting with 8c844cc82805fa33a2ada5a33f7179fc27dd0b74124a182e35a76db1201c0427 not found: ID does not exist" Aug 01 01:03:29.484695 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:29.484672 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs"] Aug 01 01:03:29.489108 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:29.489084 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b7da259d-openshift-default-9bdc59d8d-qpphs"] Aug 01 01:03:29.728748 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:29.728656 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" path="/var/lib/kubelet/pods/1758926f-9e70-468f-bfcb-2ed40e791778/volumes" Aug 01 01:03:31.811342 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:31.811306 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 01:03:31.814778 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:31.814755 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 01:03:31.817013 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:31.816993 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 01:03:31.820188 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:03:31.820168 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 01:04:04.208724 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.208690 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb"] Aug 01 01:04:04.209366 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.208875 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" podUID="e17f5cdc-37d0-47b2-8caa-4b704458feb9" containerName="manager" containerID="cri-o://a6452df031e4b766ef60e6c1160664d08f2234d0f568946772f493abc797e955" gracePeriod=10 Aug 01 01:04:04.457820 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.457794 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" Aug 01 01:04:04.513755 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.513661 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb6rf\" (UniqueName: \"kubernetes.io/projected/e17f5cdc-37d0-47b2-8caa-4b704458feb9-kube-api-access-nb6rf\") pod \"e17f5cdc-37d0-47b2-8caa-4b704458feb9\" (UID: \"e17f5cdc-37d0-47b2-8caa-4b704458feb9\") " Aug 01 01:04:04.513755 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.513753 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e17f5cdc-37d0-47b2-8caa-4b704458feb9-extensions-socket-volume\") pod \"e17f5cdc-37d0-47b2-8caa-4b704458feb9\" (UID: \"e17f5cdc-37d0-47b2-8caa-4b704458feb9\") " Aug 01 01:04:04.514218 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.514188 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17f5cdc-37d0-47b2-8caa-4b704458feb9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e17f5cdc-37d0-47b2-8caa-4b704458feb9" (UID: "e17f5cdc-37d0-47b2-8caa-4b704458feb9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:04:04.515651 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.515620 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17f5cdc-37d0-47b2-8caa-4b704458feb9-kube-api-access-nb6rf" (OuterVolumeSpecName: "kube-api-access-nb6rf") pod "e17f5cdc-37d0-47b2-8caa-4b704458feb9" (UID: "e17f5cdc-37d0-47b2-8caa-4b704458feb9"). InnerVolumeSpecName "kube-api-access-nb6rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:04:04.614654 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.614618 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e17f5cdc-37d0-47b2-8caa-4b704458feb9-extensions-socket-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:04:04.614654 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.614649 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb6rf\" (UniqueName: \"kubernetes.io/projected/e17f5cdc-37d0-47b2-8caa-4b704458feb9-kube-api-access-nb6rf\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:04:04.633719 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.633688 2563 generic.go:356] "Generic (PLEG): container finished" podID="e17f5cdc-37d0-47b2-8caa-4b704458feb9" containerID="a6452df031e4b766ef60e6c1160664d08f2234d0f568946772f493abc797e955" exitCode=0 Aug 01 01:04:04.633871 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.633735 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" Aug 01 01:04:04.633871 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.633742 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" event={"ID":"e17f5cdc-37d0-47b2-8caa-4b704458feb9","Type":"ContainerDied","Data":"a6452df031e4b766ef60e6c1160664d08f2234d0f568946772f493abc797e955"} Aug 01 01:04:04.633871 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.633785 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb" event={"ID":"e17f5cdc-37d0-47b2-8caa-4b704458feb9","Type":"ContainerDied","Data":"dd89afcbac4299a67200c9735df704108165a63a1d0c54a95ac8d252852a5f15"} Aug 01 01:04:04.633871 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.633806 2563 scope.go:117] "RemoveContainer" containerID="a6452df031e4b766ef60e6c1160664d08f2234d0f568946772f493abc797e955" Aug 01 01:04:04.643811 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.643793 2563 scope.go:117] "RemoveContainer" containerID="a6452df031e4b766ef60e6c1160664d08f2234d0f568946772f493abc797e955" Aug 01 01:04:04.644070 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:04:04.644037 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6452df031e4b766ef60e6c1160664d08f2234d0f568946772f493abc797e955\": container with ID starting with a6452df031e4b766ef60e6c1160664d08f2234d0f568946772f493abc797e955 not found: ID does not exist" containerID="a6452df031e4b766ef60e6c1160664d08f2234d0f568946772f493abc797e955" Aug 01 01:04:04.644132 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.644079 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6452df031e4b766ef60e6c1160664d08f2234d0f568946772f493abc797e955"} err="failed to get container status \"a6452df031e4b766ef60e6c1160664d08f2234d0f568946772f493abc797e955\": rpc error: code = NotFound desc = could not find container \"a6452df031e4b766ef60e6c1160664d08f2234d0f568946772f493abc797e955\": container with ID starting with a6452df031e4b766ef60e6c1160664d08f2234d0f568946772f493abc797e955 not found: ID does not exist" Aug 01 01:04:04.655686 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.655663 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb"] Aug 01 01:04:04.663398 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:04.663377 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qm8wb"] Aug 01 01:04:05.728363 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:04:05.728332 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e17f5cdc-37d0-47b2-8caa-4b704458feb9" path="/var/lib/kubelet/pods/e17f5cdc-37d0-47b2-8caa-4b704458feb9/volumes" Aug 01 01:05:19.819596 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.819563 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj"] Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.819887 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72d880e7-99e4-432e-9f94-be27217d7aa6" containerName="payload-pre-processing" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.819897 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="72d880e7-99e4-432e-9f94-be27217d7aa6" containerName="payload-pre-processing" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.819909 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e17f5cdc-37d0-47b2-8caa-4b704458feb9" containerName="manager" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.819914 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e17f5cdc-37d0-47b2-8caa-4b704458feb9" containerName="manager" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.819924 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.819930 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.819942 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96fdba51-bf80-424b-8d22-907eac7a700a" containerName="payload-processing" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.819948 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="96fdba51-bf80-424b-8d22-907eac7a700a" containerName="payload-processing" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.819956 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95556b5b-c948-4f44-8d6e-943934066a59" containerName="maas-api" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.819962 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="95556b5b-c948-4f44-8d6e-943934066a59" containerName="maas-api" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.820014 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="96fdba51-bf80-424b-8d22-907eac7a700a" containerName="payload-processing" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.820022 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="95556b5b-c948-4f44-8d6e-943934066a59" containerName="maas-api" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.820030 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.820035 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="72d880e7-99e4-432e-9f94-be27217d7aa6" containerName="payload-pre-processing" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.820041 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" Aug 01 01:05:19.820085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.820046 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e17f5cdc-37d0-47b2-8caa-4b704458feb9" containerName="manager" Aug 01 01:05:19.823091 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.823052 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj" Aug 01 01:05:19.825910 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.825886 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 01 01:05:19.826047 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.825886 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 01 01:05:19.826047 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.825891 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dnhvv\"" Aug 01 01:05:19.837349 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.837290 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj"] Aug 01 01:05:19.893419 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.893390 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c51dd1bf-0657-4eb0-a8ee-53581202dc5f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-cz2lj\" (UID: \"c51dd1bf-0657-4eb0-a8ee-53581202dc5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj" Aug 01 01:05:19.893591 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.893425 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ccxb\" (UniqueName: \"kubernetes.io/projected/c51dd1bf-0657-4eb0-a8ee-53581202dc5f-kube-api-access-5ccxb\") pod \"kuadrant-operator-controller-manager-68cf589589-cz2lj\" (UID: \"c51dd1bf-0657-4eb0-a8ee-53581202dc5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj" Aug 01 01:05:19.994237 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.994202 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c51dd1bf-0657-4eb0-a8ee-53581202dc5f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-cz2lj\" (UID: \"c51dd1bf-0657-4eb0-a8ee-53581202dc5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj" Aug 01 01:05:19.994444 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.994250 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ccxb\" (UniqueName: \"kubernetes.io/projected/c51dd1bf-0657-4eb0-a8ee-53581202dc5f-kube-api-access-5ccxb\") pod \"kuadrant-operator-controller-manager-68cf589589-cz2lj\" (UID: \"c51dd1bf-0657-4eb0-a8ee-53581202dc5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj" Aug 01 01:05:19.994604 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:19.994582 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c51dd1bf-0657-4eb0-a8ee-53581202dc5f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-cz2lj\" (UID: \"c51dd1bf-0657-4eb0-a8ee-53581202dc5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj" Aug 01 01:05:20.004240 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:20.004215 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ccxb\" (UniqueName: \"kubernetes.io/projected/c51dd1bf-0657-4eb0-a8ee-53581202dc5f-kube-api-access-5ccxb\") pod \"kuadrant-operator-controller-manager-68cf589589-cz2lj\" (UID: \"c51dd1bf-0657-4eb0-a8ee-53581202dc5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj" Aug 01 01:05:20.132941 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:20.132850 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj" Aug 01 01:05:20.262265 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:20.262236 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj"] Aug 01 01:05:20.264534 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:05:20.264510 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51dd1bf_0657_4eb0_a8ee_53581202dc5f.slice/crio-8ca40f052297e366b956557146a8e6323d14a655e191f8819d9b1d1af61ae846 WatchSource:0}: Error finding container 8ca40f052297e366b956557146a8e6323d14a655e191f8819d9b1d1af61ae846: Status 404 returned error can't find the container with id 8ca40f052297e366b956557146a8e6323d14a655e191f8819d9b1d1af61ae846 Aug 01 01:05:20.982482 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:20.982443 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj" event={"ID":"c51dd1bf-0657-4eb0-a8ee-53581202dc5f","Type":"ContainerStarted","Data":"a4b42d50a71e5ee0361b0e553c6e088e02cfa09b6811938a76260abd7bac7ba5"} Aug 01 01:05:20.982909 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:20.982501 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj" event={"ID":"c51dd1bf-0657-4eb0-a8ee-53581202dc5f","Type":"ContainerStarted","Data":"8ca40f052297e366b956557146a8e6323d14a655e191f8819d9b1d1af61ae846"} Aug 01 01:05:20.982909 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:20.982626 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj" Aug 01 01:05:21.002302 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:21.002180 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj" podStartSLOduration=2.002162146 podStartE2EDuration="2.002162146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:05:21.000825336 +0000 UTC m=+2209.789547163" watchObservedRunningTime="2026-08-01 01:05:21.002162146 +0000 UTC m=+2209.790883973" Aug 01 01:05:31.989365 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:31.989332 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cz2lj" Aug 01 01:05:38.717764 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.717721 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj"] Aug 01 01:05:38.718358 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.718284 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" Aug 01 01:05:38.718358 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.718305 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1758926f-9e70-468f-bfcb-2ed40e791778" containerName="istio-proxy" Aug 01 01:05:38.721767 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.721743 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.724409 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.724375 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e23e15a8-openshift-default-dockercfg-w7tg2\"" Aug 01 01:05:38.736024 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.735999 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj"] Aug 01 01:05:38.875912 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.875877 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j544\" (UniqueName: \"kubernetes.io/projected/b8c468f6-2855-45c0-980b-135a0db82d04-kube-api-access-8j544\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.875912 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.875927 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c468f6-2855-45c0-980b-135a0db82d04-istio-podinfo\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.876198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.875952 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c468f6-2855-45c0-980b-135a0db82d04-istiod-ca-cert\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.876198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.876007 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-workload-certs\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.876198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.876029 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-istio-data\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.876198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.876051 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c468f6-2855-45c0-980b-135a0db82d04-istio-token\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.876198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.876103 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-istio-envoy\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.876198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.876128 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-credential-socket\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.876198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.876152 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-workload-socket\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.977110 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.976999 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c468f6-2855-45c0-980b-135a0db82d04-istio-token\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.977110 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.977035 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-istio-envoy\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.977110 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.977074 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-credential-socket\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.977110 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.977101 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-workload-socket\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.977453 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.977130 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j544\" (UniqueName: \"kubernetes.io/projected/b8c468f6-2855-45c0-980b-135a0db82d04-kube-api-access-8j544\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.977453 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.977160 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c468f6-2855-45c0-980b-135a0db82d04-istio-podinfo\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.977453 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.977181 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c468f6-2855-45c0-980b-135a0db82d04-istiod-ca-cert\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.977453 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.977222 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-workload-certs\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.977453 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.977256 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-istio-data\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.977734 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.977591 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-workload-socket\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.977734 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.977661 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-istio-data\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.977734 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.977700 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-workload-certs\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.977980 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.977957 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-credential-socket\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.978040 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.977965 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c468f6-2855-45c0-980b-135a0db82d04-istiod-ca-cert\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.979436 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.979416 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-istio-envoy\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.979724 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.979705 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c468f6-2855-45c0-980b-135a0db82d04-istio-podinfo\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.986133 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.986102 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c468f6-2855-45c0-980b-135a0db82d04-istio-token\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:38.986326 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:38.986307 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j544\" (UniqueName: \"kubernetes.io/projected/b8c468f6-2855-45c0-980b-135a0db82d04-kube-api-access-8j544\") pod \"e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:39.032465 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:39.032434 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:39.175666 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:39.175639 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj"] Aug 01 01:05:39.177524 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:05:39.177500 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c468f6_2855_45c0_980b_135a0db82d04.slice/crio-06d2fa669cc988dcbdb249d49ea52d17312cae1b90a8e5e40dd9bd9ac4a2a4ed WatchSource:0}: Error finding container 06d2fa669cc988dcbdb249d49ea52d17312cae1b90a8e5e40dd9bd9ac4a2a4ed: Status 404 returned error can't find the container with id 06d2fa669cc988dcbdb249d49ea52d17312cae1b90a8e5e40dd9bd9ac4a2a4ed Aug 01 01:05:39.179562 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:39.179529 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:05:39.179631 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:39.179596 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:05:39.179668 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:39.179629 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:05:40.074154 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:40.074114 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" event={"ID":"b8c468f6-2855-45c0-980b-135a0db82d04","Type":"ContainerStarted","Data":"63265e217e48d220ec18d14d8d7f3649f4f5db60ba4310629066a6250958ebf8"} Aug 01 01:05:40.074551 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:40.074180 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" event={"ID":"b8c468f6-2855-45c0-980b-135a0db82d04","Type":"ContainerStarted","Data":"06d2fa669cc988dcbdb249d49ea52d17312cae1b90a8e5e40dd9bd9ac4a2a4ed"} Aug 01 01:05:40.094213 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:40.094151 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" podStartSLOduration=2.094129252 podStartE2EDuration="2.094129252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:05:40.093129012 +0000 UTC m=+2228.881850841" watchObservedRunningTime="2026-08-01 01:05:40.094129252 +0000 UTC m=+2228.882851079" Aug 01 01:05:41.032891 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:41.032835 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:41.039724 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:41.039694 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:41.079186 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:41.079158 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:41.080277 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:41.080254 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:43.055366 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.055332 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk"] Aug 01 01:05:43.058902 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.058880 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:05:43.061689 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.061656 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 01 01:05:43.061941 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.061906 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gf2hx\"" Aug 01 01:05:43.062676 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.062653 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e23e15a8-serving-cert\"" Aug 01 01:05:43.062821 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.062793 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:05:43.070354 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.070328 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk"] Aug 01 01:05:43.218753 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.218709 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1196a223-e4b8-4f34-b63e-a245f59f2c20-maas-api-tls\") pod \"maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk\" (UID: \"1196a223-e4b8-4f34-b63e-a245f59f2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:05:43.218973 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.218806 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1196a223-e4b8-4f34-b63e-a245f59f2c20-openshift-service-ca\") pod \"maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk\" (UID: \"1196a223-e4b8-4f34-b63e-a245f59f2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:05:43.218973 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.218844 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pszr7\" (UniqueName: \"kubernetes.io/projected/1196a223-e4b8-4f34-b63e-a245f59f2c20-kube-api-access-pszr7\") pod \"maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk\" (UID: \"1196a223-e4b8-4f34-b63e-a245f59f2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:05:43.320414 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.320323 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1196a223-e4b8-4f34-b63e-a245f59f2c20-openshift-service-ca\") pod \"maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk\" (UID: \"1196a223-e4b8-4f34-b63e-a245f59f2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:05:43.320414 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.320366 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pszr7\" (UniqueName: \"kubernetes.io/projected/1196a223-e4b8-4f34-b63e-a245f59f2c20-kube-api-access-pszr7\") pod \"maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk\" (UID: \"1196a223-e4b8-4f34-b63e-a245f59f2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:05:43.320612 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.320451 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1196a223-e4b8-4f34-b63e-a245f59f2c20-maas-api-tls\") pod \"maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk\" (UID: \"1196a223-e4b8-4f34-b63e-a245f59f2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:05:43.321040 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.321016 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1196a223-e4b8-4f34-b63e-a245f59f2c20-openshift-service-ca\") pod \"maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk\" (UID: \"1196a223-e4b8-4f34-b63e-a245f59f2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:05:43.322937 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.322910 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1196a223-e4b8-4f34-b63e-a245f59f2c20-maas-api-tls\") pod \"maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk\" (UID: \"1196a223-e4b8-4f34-b63e-a245f59f2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:05:43.328341 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.328321 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pszr7\" (UniqueName: \"kubernetes.io/projected/1196a223-e4b8-4f34-b63e-a245f59f2c20-kube-api-access-pszr7\") pod \"maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk\" (UID: \"1196a223-e4b8-4f34-b63e-a245f59f2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:05:43.372325 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.372292 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:05:43.514364 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.514336 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk"] Aug 01 01:05:43.516196 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:05:43.516167 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1196a223_e4b8_4f34_b63e_a245f59f2c20.slice/crio-75b468b5112e122b649f0575089b9bacbf93284fbf920b18397800742bc6a595 WatchSource:0}: Error finding container 75b468b5112e122b649f0575089b9bacbf93284fbf920b18397800742bc6a595: Status 404 returned error can't find the container with id 75b468b5112e122b649f0575089b9bacbf93284fbf920b18397800742bc6a595 Aug 01 01:05:43.517632 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.517615 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 01:05:43.707015 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.706932 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq"] Aug 01 01:05:43.711773 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.711747 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" Aug 01 01:05:43.714500 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.714478 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e23e15a8-dockercfg-d894l\"" Aug 01 01:05:43.714663 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.714643 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e23e15a8\"" Aug 01 01:05:43.720765 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.720723 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq"] Aug 01 01:05:43.730619 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.730593 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd"] Aug 01 01:05:43.734002 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.733984 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" Aug 01 01:05:43.743928 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.743900 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd"] Aug 01 01:05:43.825985 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.825945 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtm2\" (UniqueName: \"kubernetes.io/projected/95c6ed20-3267-452c-bacc-0b559502e88a-kube-api-access-vvtm2\") pod \"payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd\" (UID: \"95c6ed20-3267-452c-bacc-0b559502e88a\") " pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" Aug 01 01:05:43.826203 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.826078 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6053929-2c77-4359-86c7-f50409b503af-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq\" (UID: \"c6053929-2c77-4359-86c7-f50409b503af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" Aug 01 01:05:43.826203 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.826187 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-269ls\" (UniqueName: \"kubernetes.io/projected/c6053929-2c77-4359-86c7-f50409b503af-kube-api-access-269ls\") pod \"payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq\" (UID: \"c6053929-2c77-4359-86c7-f50409b503af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" Aug 01 01:05:43.826330 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.826250 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c6ed20-3267-452c-bacc-0b559502e88a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd\" (UID: \"95c6ed20-3267-452c-bacc-0b559502e88a\") " pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" Aug 01 01:05:43.927202 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.927162 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtm2\" (UniqueName: \"kubernetes.io/projected/95c6ed20-3267-452c-bacc-0b559502e88a-kube-api-access-vvtm2\") pod \"payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd\" (UID: \"95c6ed20-3267-452c-bacc-0b559502e88a\") " pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" Aug 01 01:05:43.927386 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.927237 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6053929-2c77-4359-86c7-f50409b503af-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq\" (UID: \"c6053929-2c77-4359-86c7-f50409b503af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" Aug 01 01:05:43.927386 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.927332 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-269ls\" (UniqueName: \"kubernetes.io/projected/c6053929-2c77-4359-86c7-f50409b503af-kube-api-access-269ls\") pod \"payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq\" (UID: \"c6053929-2c77-4359-86c7-f50409b503af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" Aug 01 01:05:43.927510 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.927388 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c6ed20-3267-452c-bacc-0b559502e88a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd\" (UID: \"95c6ed20-3267-452c-bacc-0b559502e88a\") " pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" Aug 01 01:05:43.928035 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.928011 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6053929-2c77-4359-86c7-f50409b503af-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq\" (UID: \"c6053929-2c77-4359-86c7-f50409b503af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" Aug 01 01:05:43.928246 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.928224 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c6ed20-3267-452c-bacc-0b559502e88a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd\" (UID: \"95c6ed20-3267-452c-bacc-0b559502e88a\") " pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" Aug 01 01:05:43.935610 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.935580 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-269ls\" (UniqueName: \"kubernetes.io/projected/c6053929-2c77-4359-86c7-f50409b503af-kube-api-access-269ls\") pod \"payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq\" (UID: \"c6053929-2c77-4359-86c7-f50409b503af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" Aug 01 01:05:43.935710 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:43.935610 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtm2\" (UniqueName: \"kubernetes.io/projected/95c6ed20-3267-452c-bacc-0b559502e88a-kube-api-access-vvtm2\") pod \"payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd\" (UID: \"95c6ed20-3267-452c-bacc-0b559502e88a\") " pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" Aug 01 01:05:44.026980 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:44.026887 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" Aug 01 01:05:44.046825 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:44.046782 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" Aug 01 01:05:44.097716 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:44.097681 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" event={"ID":"1196a223-e4b8-4f34-b63e-a245f59f2c20","Type":"ContainerStarted","Data":"75b468b5112e122b649f0575089b9bacbf93284fbf920b18397800742bc6a595"} Aug 01 01:05:44.207506 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:44.207477 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq"] Aug 01 01:05:44.209264 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:05:44.209237 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6053929_2c77_4359_86c7_f50409b503af.slice/crio-baf9bcaa93f5ee9422c10e0d3ca956683130d6573e22cf4bf622eb9779aecc11 WatchSource:0}: Error finding container baf9bcaa93f5ee9422c10e0d3ca956683130d6573e22cf4bf622eb9779aecc11: Status 404 returned error can't find the container with id baf9bcaa93f5ee9422c10e0d3ca956683130d6573e22cf4bf622eb9779aecc11 Aug 01 01:05:44.227941 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:44.227895 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd"] Aug 01 01:05:44.229686 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:05:44.229659 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c6ed20_3267_452c_bacc_0b559502e88a.slice/crio-1ea49ed70a28f81fce561a87573be0ad22b8ac9a7cf5728e9c6c49caa8113430 WatchSource:0}: Error finding container 1ea49ed70a28f81fce561a87573be0ad22b8ac9a7cf5728e9c6c49caa8113430: Status 404 returned error can't find the container with id 1ea49ed70a28f81fce561a87573be0ad22b8ac9a7cf5728e9c6c49caa8113430 Aug 01 01:05:45.103638 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:45.103600 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" event={"ID":"1196a223-e4b8-4f34-b63e-a245f59f2c20","Type":"ContainerStarted","Data":"d55e4f900f81cb3cb68fa38703e50d359fbf62520eb210b68538ef095bcd357a"} Aug 01 01:05:45.104169 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:45.103760 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:05:45.105222 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:45.105202 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" event={"ID":"95c6ed20-3267-452c-bacc-0b559502e88a","Type":"ContainerStarted","Data":"b926c009fe1e64fcc36b9143afccabda8280c9891628392495efa6190e2a901e"} Aug 01 01:05:45.105309 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:45.105244 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" Aug 01 01:05:45.105309 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:45.105267 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" event={"ID":"95c6ed20-3267-452c-bacc-0b559502e88a","Type":"ContainerStarted","Data":"1ea49ed70a28f81fce561a87573be0ad22b8ac9a7cf5728e9c6c49caa8113430"} Aug 01 01:05:45.106678 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:45.106654 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" event={"ID":"c6053929-2c77-4359-86c7-f50409b503af","Type":"ContainerStarted","Data":"6abaa470f4c38df35b5298a53135c81655b7fab7b9cacf5cdb69db3ed2ac4cee"} Aug 01 01:05:45.106781 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:45.106698 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" event={"ID":"c6053929-2c77-4359-86c7-f50409b503af","Type":"ContainerStarted","Data":"baf9bcaa93f5ee9422c10e0d3ca956683130d6573e22cf4bf622eb9779aecc11"} Aug 01 01:05:45.106843 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:45.106789 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" Aug 01 01:05:45.121621 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:45.121556 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" podStartSLOduration=1.463691589 podStartE2EDuration="2.121532846s" totalImagesPullingTime="657.841257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:05:45.119396707 +0000 UTC m=+2233.908118534" watchObservedRunningTime="2026-08-01 01:05:45.121532846 +0000 UTC m=+2233.910254674" Aug 01 01:05:45.139760 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:45.139705 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" podStartSLOduration=2.139687887 podStartE2EDuration="2.139687887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:05:45.137902829 +0000 UTC m=+2233.926624689" watchObservedRunningTime="2026-08-01 01:05:45.139687887 +0000 UTC m=+2233.928409713" Aug 01 01:05:45.153731 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:45.153682 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" podStartSLOduration=2.153665601 podStartE2EDuration="2.153665601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:05:45.152018132 +0000 UTC m=+2233.940739959" watchObservedRunningTime="2026-08-01 01:05:45.153665601 +0000 UTC m=+2233.942387426" Aug 01 01:05:51.126359 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:51.125377 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:05:55.168544 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:55.168465 2563 generic.go:356] "Generic (PLEG): container finished" podID="b8c468f6-2855-45c0-980b-135a0db82d04" containerID="63265e217e48d220ec18d14d8d7f3649f4f5db60ba4310629066a6250958ebf8" exitCode=0 Aug 01 01:05:55.168921 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:55.168533 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" event={"ID":"b8c468f6-2855-45c0-980b-135a0db82d04","Type":"ContainerDied","Data":"63265e217e48d220ec18d14d8d7f3649f4f5db60ba4310629066a6250958ebf8"} Aug 01 01:05:55.168976 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:55.168949 2563 scope.go:117] "RemoveContainer" containerID="63265e217e48d220ec18d14d8d7f3649f4f5db60ba4310629066a6250958ebf8" Aug 01 01:05:55.169952 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:55.169926 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:05:55.170004 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:55.169991 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:05:55.170038 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:55.170025 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:05:56.079338 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:56.079292 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:56.116291 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:56.116257 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" Aug 01 01:05:56.116644 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:56.116622 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" Aug 01 01:05:56.176834 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:56.176796 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" event={"ID":"b8c468f6-2855-45c0-980b-135a0db82d04","Type":"ContainerStarted","Data":"3fdfb448fbf3bee20787b1b9973da3194447db1a8c31c3d05164022cca4ebad3"} Aug 01 01:05:57.033799 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:57.033386 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:05:57.034962 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:57.034931 2563 patch_prober.go:28] interesting pod/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Aug 01 01:05:57.035209 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:57.035178 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" podUID="b8c468f6-2855-45c0-980b-135a0db82d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Aug 01 01:05:58.032948 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:58.032904 2563 patch_prober.go:28] interesting pod/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Aug 01 01:05:58.033465 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:58.032977 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" podUID="b8c468f6-2855-45c0-980b-135a0db82d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Aug 01 01:05:59.033013 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:59.032968 2563 patch_prober.go:28] interesting pod/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Aug 01 01:05:59.033634 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:05:59.033602 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" podUID="b8c468f6-2855-45c0-980b-135a0db82d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Aug 01 01:06:00.033305 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:06:00.033264 2563 patch_prober.go:28] interesting pod/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Aug 01 01:06:00.033918 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:06:00.033884 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" podUID="b8c468f6-2855-45c0-980b-135a0db82d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Aug 01 01:06:01.032964 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:06:01.032913 2563 patch_prober.go:28] interesting pod/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Aug 01 01:06:01.033200 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:06:01.032989 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" podUID="b8c468f6-2855-45c0-980b-135a0db82d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Aug 01 01:06:02.037167 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:06:02.037133 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:06:02.037528 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:06:02.037406 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:06:02.038005 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:06:02.037991 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:06:31.937277 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:06:31.937234 2563 scope.go:117] "RemoveContainer" containerID="d31ed5913949d10d9ca84ce7cada3c1957d947eb9e98cf0655699be63bb8533b" Aug 01 01:07:01.802328 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:01.802295 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk"] Aug 01 01:07:01.802885 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:01.802802 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" podUID="1196a223-e4b8-4f34-b63e-a245f59f2c20" containerName="maas-api" containerID="cri-o://d55e4f900f81cb3cb68fa38703e50d359fbf62520eb210b68538ef095bcd357a" gracePeriod=30 Aug 01 01:07:01.859203 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:01.859173 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd"] Aug 01 01:07:01.859569 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:01.859414 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" podUID="95c6ed20-3267-452c-bacc-0b559502e88a" containerName="payload-processing" containerID="cri-o://b926c009fe1e64fcc36b9143afccabda8280c9891628392495efa6190e2a901e" gracePeriod=30 Aug 01 01:07:01.865341 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:01.865316 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq"] Aug 01 01:07:01.865586 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:01.865524 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" podUID="c6053929-2c77-4359-86c7-f50409b503af" containerName="payload-pre-processing" containerID="cri-o://6abaa470f4c38df35b5298a53135c81655b7fab7b9cacf5cdb69db3ed2ac4cee" gracePeriod=30 Aug 01 01:07:02.151023 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.150995 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" Aug 01 01:07:02.153942 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.153922 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:07:02.166724 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.166706 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" Aug 01 01:07:02.255327 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.255291 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1196a223-e4b8-4f34-b63e-a245f59f2c20-maas-api-tls\") pod \"1196a223-e4b8-4f34-b63e-a245f59f2c20\" (UID: \"1196a223-e4b8-4f34-b63e-a245f59f2c20\") " Aug 01 01:07:02.255495 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.255338 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1196a223-e4b8-4f34-b63e-a245f59f2c20-openshift-service-ca\") pod \"1196a223-e4b8-4f34-b63e-a245f59f2c20\" (UID: \"1196a223-e4b8-4f34-b63e-a245f59f2c20\") " Aug 01 01:07:02.255495 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.255360 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6053929-2c77-4359-86c7-f50409b503af-plugins-config-volume\") pod \"c6053929-2c77-4359-86c7-f50409b503af\" (UID: \"c6053929-2c77-4359-86c7-f50409b503af\") " Aug 01 01:07:02.255495 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.255382 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c6ed20-3267-452c-bacc-0b559502e88a-plugins-config-volume\") pod \"95c6ed20-3267-452c-bacc-0b559502e88a\" (UID: \"95c6ed20-3267-452c-bacc-0b559502e88a\") " Aug 01 01:07:02.255495 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.255414 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pszr7\" (UniqueName: \"kubernetes.io/projected/1196a223-e4b8-4f34-b63e-a245f59f2c20-kube-api-access-pszr7\") pod \"1196a223-e4b8-4f34-b63e-a245f59f2c20\" (UID: \"1196a223-e4b8-4f34-b63e-a245f59f2c20\") " Aug 01 01:07:02.255495 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.255434 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-269ls\" (UniqueName: \"kubernetes.io/projected/c6053929-2c77-4359-86c7-f50409b503af-kube-api-access-269ls\") pod \"c6053929-2c77-4359-86c7-f50409b503af\" (UID: \"c6053929-2c77-4359-86c7-f50409b503af\") " Aug 01 01:07:02.255777 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.255539 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvtm2\" (UniqueName: \"kubernetes.io/projected/95c6ed20-3267-452c-bacc-0b559502e88a-kube-api-access-vvtm2\") pod \"95c6ed20-3267-452c-bacc-0b559502e88a\" (UID: \"95c6ed20-3267-452c-bacc-0b559502e88a\") " Aug 01 01:07:02.255777 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.255732 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1196a223-e4b8-4f34-b63e-a245f59f2c20-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1196a223-e4b8-4f34-b63e-a245f59f2c20" (UID: "1196a223-e4b8-4f34-b63e-a245f59f2c20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:07:02.255777 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.255754 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6053929-2c77-4359-86c7-f50409b503af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6053929-2c77-4359-86c7-f50409b503af" (UID: "c6053929-2c77-4359-86c7-f50409b503af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:07:02.255950 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.255782 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c6ed20-3267-452c-bacc-0b559502e88a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95c6ed20-3267-452c-bacc-0b559502e88a" (UID: "95c6ed20-3267-452c-bacc-0b559502e88a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:07:02.255950 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.255882 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1196a223-e4b8-4f34-b63e-a245f59f2c20-openshift-service-ca\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:02.255950 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.255900 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6053929-2c77-4359-86c7-f50409b503af-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:02.255950 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.255913 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c6ed20-3267-452c-bacc-0b559502e88a-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:02.257602 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.257580 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1196a223-e4b8-4f34-b63e-a245f59f2c20-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1196a223-e4b8-4f34-b63e-a245f59f2c20" (UID: "1196a223-e4b8-4f34-b63e-a245f59f2c20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:07:02.257602 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.257587 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6053929-2c77-4359-86c7-f50409b503af-kube-api-access-269ls" (OuterVolumeSpecName: "kube-api-access-269ls") pod "c6053929-2c77-4359-86c7-f50409b503af" (UID: "c6053929-2c77-4359-86c7-f50409b503af"). InnerVolumeSpecName "kube-api-access-269ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:07:02.257751 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.257727 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1196a223-e4b8-4f34-b63e-a245f59f2c20-kube-api-access-pszr7" (OuterVolumeSpecName: "kube-api-access-pszr7") pod "1196a223-e4b8-4f34-b63e-a245f59f2c20" (UID: "1196a223-e4b8-4f34-b63e-a245f59f2c20"). InnerVolumeSpecName "kube-api-access-pszr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:07:02.257795 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.257779 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c6ed20-3267-452c-bacc-0b559502e88a-kube-api-access-vvtm2" (OuterVolumeSpecName: "kube-api-access-vvtm2") pod "95c6ed20-3267-452c-bacc-0b559502e88a" (UID: "95c6ed20-3267-452c-bacc-0b559502e88a"). InnerVolumeSpecName "kube-api-access-vvtm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:07:02.356548 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.356473 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvtm2\" (UniqueName: \"kubernetes.io/projected/95c6ed20-3267-452c-bacc-0b559502e88a-kube-api-access-vvtm2\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:02.356548 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.356500 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1196a223-e4b8-4f34-b63e-a245f59f2c20-maas-api-tls\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:02.356548 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.356509 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pszr7\" (UniqueName: \"kubernetes.io/projected/1196a223-e4b8-4f34-b63e-a245f59f2c20-kube-api-access-pszr7\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:02.356548 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.356518 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-269ls\" (UniqueName: \"kubernetes.io/projected/c6053929-2c77-4359-86c7-f50409b503af-kube-api-access-269ls\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:02.497789 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.497754 2563 generic.go:356] "Generic (PLEG): container finished" podID="95c6ed20-3267-452c-bacc-0b559502e88a" containerID="b926c009fe1e64fcc36b9143afccabda8280c9891628392495efa6190e2a901e" exitCode=0 Aug 01 01:07:02.497939 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.497801 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" Aug 01 01:07:02.497939 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.497838 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" event={"ID":"95c6ed20-3267-452c-bacc-0b559502e88a","Type":"ContainerDied","Data":"b926c009fe1e64fcc36b9143afccabda8280c9891628392495efa6190e2a901e"} Aug 01 01:07:02.497939 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.497900 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd" event={"ID":"95c6ed20-3267-452c-bacc-0b559502e88a","Type":"ContainerDied","Data":"1ea49ed70a28f81fce561a87573be0ad22b8ac9a7cf5728e9c6c49caa8113430"} Aug 01 01:07:02.497939 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.497930 2563 scope.go:117] "RemoveContainer" containerID="b926c009fe1e64fcc36b9143afccabda8280c9891628392495efa6190e2a901e" Aug 01 01:07:02.499468 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.499437 2563 generic.go:356] "Generic (PLEG): container finished" podID="c6053929-2c77-4359-86c7-f50409b503af" containerID="6abaa470f4c38df35b5298a53135c81655b7fab7b9cacf5cdb69db3ed2ac4cee" exitCode=0 Aug 01 01:07:02.499601 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.499496 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" Aug 01 01:07:02.499601 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.499523 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" event={"ID":"c6053929-2c77-4359-86c7-f50409b503af","Type":"ContainerDied","Data":"6abaa470f4c38df35b5298a53135c81655b7fab7b9cacf5cdb69db3ed2ac4cee"} Aug 01 01:07:02.499601 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.499588 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq" event={"ID":"c6053929-2c77-4359-86c7-f50409b503af","Type":"ContainerDied","Data":"baf9bcaa93f5ee9422c10e0d3ca956683130d6573e22cf4bf622eb9779aecc11"} Aug 01 01:07:02.500987 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.500943 2563 generic.go:356] "Generic (PLEG): container finished" podID="1196a223-e4b8-4f34-b63e-a245f59f2c20" containerID="d55e4f900f81cb3cb68fa38703e50d359fbf62520eb210b68538ef095bcd357a" exitCode=0 Aug 01 01:07:02.500987 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.500977 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" event={"ID":"1196a223-e4b8-4f34-b63e-a245f59f2c20","Type":"ContainerDied","Data":"d55e4f900f81cb3cb68fa38703e50d359fbf62520eb210b68538ef095bcd357a"} Aug 01 01:07:02.501165 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.501015 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" event={"ID":"1196a223-e4b8-4f34-b63e-a245f59f2c20","Type":"ContainerDied","Data":"75b468b5112e122b649f0575089b9bacbf93284fbf920b18397800742bc6a595"} Aug 01 01:07:02.501165 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.501078 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk" Aug 01 01:07:02.507598 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.507567 2563 scope.go:117] "RemoveContainer" containerID="b926c009fe1e64fcc36b9143afccabda8280c9891628392495efa6190e2a901e" Aug 01 01:07:02.507908 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:07:02.507879 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b926c009fe1e64fcc36b9143afccabda8280c9891628392495efa6190e2a901e\": container with ID starting with b926c009fe1e64fcc36b9143afccabda8280c9891628392495efa6190e2a901e not found: ID does not exist" containerID="b926c009fe1e64fcc36b9143afccabda8280c9891628392495efa6190e2a901e" Aug 01 01:07:02.507991 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.507918 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b926c009fe1e64fcc36b9143afccabda8280c9891628392495efa6190e2a901e"} err="failed to get container status \"b926c009fe1e64fcc36b9143afccabda8280c9891628392495efa6190e2a901e\": rpc error: code = NotFound desc = could not find container \"b926c009fe1e64fcc36b9143afccabda8280c9891628392495efa6190e2a901e\": container with ID starting with b926c009fe1e64fcc36b9143afccabda8280c9891628392495efa6190e2a901e not found: ID does not exist" Aug 01 01:07:02.507991 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.507943 2563 scope.go:117] "RemoveContainer" containerID="6abaa470f4c38df35b5298a53135c81655b7fab7b9cacf5cdb69db3ed2ac4cee" Aug 01 01:07:02.516246 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.516230 2563 scope.go:117] "RemoveContainer" containerID="6abaa470f4c38df35b5298a53135c81655b7fab7b9cacf5cdb69db3ed2ac4cee" Aug 01 01:07:02.516497 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:07:02.516476 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abaa470f4c38df35b5298a53135c81655b7fab7b9cacf5cdb69db3ed2ac4cee\": container with ID starting with 6abaa470f4c38df35b5298a53135c81655b7fab7b9cacf5cdb69db3ed2ac4cee not found: ID does not exist" containerID="6abaa470f4c38df35b5298a53135c81655b7fab7b9cacf5cdb69db3ed2ac4cee" Aug 01 01:07:02.516559 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.516503 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abaa470f4c38df35b5298a53135c81655b7fab7b9cacf5cdb69db3ed2ac4cee"} err="failed to get container status \"6abaa470f4c38df35b5298a53135c81655b7fab7b9cacf5cdb69db3ed2ac4cee\": rpc error: code = NotFound desc = could not find container \"6abaa470f4c38df35b5298a53135c81655b7fab7b9cacf5cdb69db3ed2ac4cee\": container with ID starting with 6abaa470f4c38df35b5298a53135c81655b7fab7b9cacf5cdb69db3ed2ac4cee not found: ID does not exist" Aug 01 01:07:02.516559 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.516518 2563 scope.go:117] "RemoveContainer" containerID="d55e4f900f81cb3cb68fa38703e50d359fbf62520eb210b68538ef095bcd357a" Aug 01 01:07:02.526156 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.526138 2563 scope.go:117] "RemoveContainer" containerID="d55e4f900f81cb3cb68fa38703e50d359fbf62520eb210b68538ef095bcd357a" Aug 01 01:07:02.526429 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:07:02.526405 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55e4f900f81cb3cb68fa38703e50d359fbf62520eb210b68538ef095bcd357a\": container with ID starting with d55e4f900f81cb3cb68fa38703e50d359fbf62520eb210b68538ef095bcd357a not found: ID does not exist" containerID="d55e4f900f81cb3cb68fa38703e50d359fbf62520eb210b68538ef095bcd357a" Aug 01 01:07:02.526504 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.526437 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55e4f900f81cb3cb68fa38703e50d359fbf62520eb210b68538ef095bcd357a"} err="failed to get container status \"d55e4f900f81cb3cb68fa38703e50d359fbf62520eb210b68538ef095bcd357a\": rpc error: code = NotFound desc = could not find container \"d55e4f900f81cb3cb68fa38703e50d359fbf62520eb210b68538ef095bcd357a\": container with ID starting with d55e4f900f81cb3cb68fa38703e50d359fbf62520eb210b68538ef095bcd357a not found: ID does not exist" Aug 01 01:07:02.526569 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.526530 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq"] Aug 01 01:07:02.531898 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.531873 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e23e15a8-7d5fc8d5c6-9hdbq"] Aug 01 01:07:02.542802 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.542780 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd"] Aug 01 01:07:02.546518 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.546500 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e23e15a8-78bfd64c54-rgdwd"] Aug 01 01:07:02.556511 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.556493 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk"] Aug 01 01:07:02.562230 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:02.562209 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e23e15a8-7f99d8c86f-xd8mk"] Aug 01 01:07:03.728466 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:03.728433 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1196a223-e4b8-4f34-b63e-a245f59f2c20" path="/var/lib/kubelet/pods/1196a223-e4b8-4f34-b63e-a245f59f2c20/volumes" Aug 01 01:07:03.728834 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:03.728800 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c6ed20-3267-452c-bacc-0b559502e88a" path="/var/lib/kubelet/pods/95c6ed20-3267-452c-bacc-0b559502e88a/volumes" Aug 01 01:07:03.729203 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:03.729185 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6053929-2c77-4359-86c7-f50409b503af" path="/var/lib/kubelet/pods/c6053929-2c77-4359-86c7-f50409b503af/volumes" Aug 01 01:07:53.021520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:53.021486 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj"] Aug 01 01:07:53.022030 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:53.021756 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" podUID="b8c468f6-2855-45c0-980b-135a0db82d04" containerName="istio-proxy" containerID="cri-o://3fdfb448fbf3bee20787b1b9973da3194447db1a8c31c3d05164022cca4ebad3" gracePeriod=30 Aug 01 01:07:58.270828 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.270804 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:07:58.449709 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.449680 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c468f6-2855-45c0-980b-135a0db82d04-istio-podinfo\") pod \"b8c468f6-2855-45c0-980b-135a0db82d04\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " Aug 01 01:07:58.449912 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.449735 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j544\" (UniqueName: \"kubernetes.io/projected/b8c468f6-2855-45c0-980b-135a0db82d04-kube-api-access-8j544\") pod \"b8c468f6-2855-45c0-980b-135a0db82d04\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " Aug 01 01:07:58.449912 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.449786 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-workload-socket\") pod \"b8c468f6-2855-45c0-980b-135a0db82d04\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " Aug 01 01:07:58.449912 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.449803 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-istio-envoy\") pod \"b8c468f6-2855-45c0-980b-135a0db82d04\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " Aug 01 01:07:58.449912 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.449859 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c468f6-2855-45c0-980b-135a0db82d04-istio-token\") pod \"b8c468f6-2855-45c0-980b-135a0db82d04\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " Aug 01 01:07:58.449912 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.449886 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-workload-certs\") pod \"b8c468f6-2855-45c0-980b-135a0db82d04\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " Aug 01 01:07:58.450218 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.449921 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c468f6-2855-45c0-980b-135a0db82d04-istiod-ca-cert\") pod \"b8c468f6-2855-45c0-980b-135a0db82d04\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " Aug 01 01:07:58.450218 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.449943 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-istio-data\") pod \"b8c468f6-2855-45c0-980b-135a0db82d04\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " Aug 01 01:07:58.450218 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.449965 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-credential-socket\") pod \"b8c468f6-2855-45c0-980b-135a0db82d04\" (UID: \"b8c468f6-2855-45c0-980b-135a0db82d04\") " Aug 01 01:07:58.450218 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.450174 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b8c468f6-2855-45c0-980b-135a0db82d04" (UID: "b8c468f6-2855-45c0-980b-135a0db82d04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:07:58.450435 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.450326 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-workload-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:58.450435 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.450373 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b8c468f6-2855-45c0-980b-135a0db82d04" (UID: "b8c468f6-2855-45c0-980b-135a0db82d04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:07:58.450534 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.450508 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-istio-data" (OuterVolumeSpecName: "istio-data") pod "b8c468f6-2855-45c0-980b-135a0db82d04" (UID: "b8c468f6-2855-45c0-980b-135a0db82d04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:07:58.450608 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.450586 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b8c468f6-2855-45c0-980b-135a0db82d04" (UID: "b8c468f6-2855-45c0-980b-135a0db82d04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:07:58.450686 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.450452 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c468f6-2855-45c0-980b-135a0db82d04-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b8c468f6-2855-45c0-980b-135a0db82d04" (UID: "b8c468f6-2855-45c0-980b-135a0db82d04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:07:58.452049 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.452011 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c468f6-2855-45c0-980b-135a0db82d04-kube-api-access-8j544" (OuterVolumeSpecName: "kube-api-access-8j544") pod "b8c468f6-2855-45c0-980b-135a0db82d04" (UID: "b8c468f6-2855-45c0-980b-135a0db82d04"). InnerVolumeSpecName "kube-api-access-8j544". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:07:58.452049 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.452051 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c468f6-2855-45c0-980b-135a0db82d04-istio-token" (OuterVolumeSpecName: "istio-token") pod "b8c468f6-2855-45c0-980b-135a0db82d04" (UID: "b8c468f6-2855-45c0-980b-135a0db82d04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:07:58.452310 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.452281 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b8c468f6-2855-45c0-980b-135a0db82d04" (UID: "b8c468f6-2855-45c0-980b-135a0db82d04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:07:58.452352 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.452293 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8c468f6-2855-45c0-980b-135a0db82d04-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b8c468f6-2855-45c0-980b-135a0db82d04" (UID: "b8c468f6-2855-45c0-980b-135a0db82d04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 01 01:07:58.551228 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.551199 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j544\" (UniqueName: \"kubernetes.io/projected/b8c468f6-2855-45c0-980b-135a0db82d04-kube-api-access-8j544\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:58.551228 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.551223 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-istio-envoy\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:58.551228 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.551234 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c468f6-2855-45c0-980b-135a0db82d04-istio-token\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:58.551443 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.551242 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-workload-certs\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:58.551443 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.551250 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c468f6-2855-45c0-980b-135a0db82d04-istiod-ca-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:58.551443 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.551257 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-istio-data\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:58.551443 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.551265 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c468f6-2855-45c0-980b-135a0db82d04-credential-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:58.551443 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.551272 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c468f6-2855-45c0-980b-135a0db82d04-istio-podinfo\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:07:58.770085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.769990 2563 generic.go:356] "Generic (PLEG): container finished" podID="b8c468f6-2855-45c0-980b-135a0db82d04" containerID="3fdfb448fbf3bee20787b1b9973da3194447db1a8c31c3d05164022cca4ebad3" exitCode=0 Aug 01 01:07:58.770085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.770038 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" Aug 01 01:07:58.770085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.770075 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" event={"ID":"b8c468f6-2855-45c0-980b-135a0db82d04","Type":"ContainerDied","Data":"3fdfb448fbf3bee20787b1b9973da3194447db1a8c31c3d05164022cca4ebad3"} Aug 01 01:07:58.770306 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.770116 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj" event={"ID":"b8c468f6-2855-45c0-980b-135a0db82d04","Type":"ContainerDied","Data":"06d2fa669cc988dcbdb249d49ea52d17312cae1b90a8e5e40dd9bd9ac4a2a4ed"} Aug 01 01:07:58.770306 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.770136 2563 scope.go:117] "RemoveContainer" containerID="3fdfb448fbf3bee20787b1b9973da3194447db1a8c31c3d05164022cca4ebad3" Aug 01 01:07:58.778811 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.778793 2563 scope.go:117] "RemoveContainer" containerID="63265e217e48d220ec18d14d8d7f3649f4f5db60ba4310629066a6250958ebf8" Aug 01 01:07:58.786238 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.786218 2563 scope.go:117] "RemoveContainer" containerID="3fdfb448fbf3bee20787b1b9973da3194447db1a8c31c3d05164022cca4ebad3" Aug 01 01:07:58.786487 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:07:58.786469 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fdfb448fbf3bee20787b1b9973da3194447db1a8c31c3d05164022cca4ebad3\": container with ID starting with 3fdfb448fbf3bee20787b1b9973da3194447db1a8c31c3d05164022cca4ebad3 not found: ID does not exist" containerID="3fdfb448fbf3bee20787b1b9973da3194447db1a8c31c3d05164022cca4ebad3" Aug 01 01:07:58.786544 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.786494 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fdfb448fbf3bee20787b1b9973da3194447db1a8c31c3d05164022cca4ebad3"} err="failed to get container status \"3fdfb448fbf3bee20787b1b9973da3194447db1a8c31c3d05164022cca4ebad3\": rpc error: code = NotFound desc = could not find container \"3fdfb448fbf3bee20787b1b9973da3194447db1a8c31c3d05164022cca4ebad3\": container with ID starting with 3fdfb448fbf3bee20787b1b9973da3194447db1a8c31c3d05164022cca4ebad3 not found: ID does not exist" Aug 01 01:07:58.786544 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.786511 2563 scope.go:117] "RemoveContainer" containerID="63265e217e48d220ec18d14d8d7f3649f4f5db60ba4310629066a6250958ebf8" Aug 01 01:07:58.786755 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:07:58.786740 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63265e217e48d220ec18d14d8d7f3649f4f5db60ba4310629066a6250958ebf8\": container with ID starting with 63265e217e48d220ec18d14d8d7f3649f4f5db60ba4310629066a6250958ebf8 not found: ID does not exist" containerID="63265e217e48d220ec18d14d8d7f3649f4f5db60ba4310629066a6250958ebf8" Aug 01 01:07:58.786807 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.786758 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63265e217e48d220ec18d14d8d7f3649f4f5db60ba4310629066a6250958ebf8"} err="failed to get container status \"63265e217e48d220ec18d14d8d7f3649f4f5db60ba4310629066a6250958ebf8\": rpc error: code = NotFound desc = could not find container \"63265e217e48d220ec18d14d8d7f3649f4f5db60ba4310629066a6250958ebf8\": container with ID starting with 63265e217e48d220ec18d14d8d7f3649f4f5db60ba4310629066a6250958ebf8 not found: ID does not exist" Aug 01 01:07:58.793350 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.793329 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj"] Aug 01 01:07:58.797452 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:58.797432 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e23e15a8-openshift-default-b78d86ffb-t45nj"] Aug 01 01:07:59.728368 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:07:59.728335 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8c468f6-2855-45c0-980b-135a0db82d04" path="/var/lib/kubelet/pods/b8c468f6-2855-45c0-980b-135a0db82d04/volumes" Aug 01 01:08:31.841470 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:31.841427 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 01:08:31.846470 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:31.846446 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 01:08:31.847628 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:31.847603 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 01:08:31.852340 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:31.852324 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 01:08:57.467807 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.467725 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn"] Aug 01 01:08:57.468320 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468207 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c468f6-2855-45c0-980b-135a0db82d04" containerName="istio-proxy" Aug 01 01:08:57.468320 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468222 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c468f6-2855-45c0-980b-135a0db82d04" containerName="istio-proxy" Aug 01 01:08:57.468320 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468234 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c468f6-2855-45c0-980b-135a0db82d04" containerName="istio-proxy" Aug 01 01:08:57.468320 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468241 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c468f6-2855-45c0-980b-135a0db82d04" containerName="istio-proxy" Aug 01 01:08:57.468320 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468253 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c6ed20-3267-452c-bacc-0b559502e88a" containerName="payload-processing" Aug 01 01:08:57.468320 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468260 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c6ed20-3267-452c-bacc-0b559502e88a" containerName="payload-processing" Aug 01 01:08:57.468320 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468272 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1196a223-e4b8-4f34-b63e-a245f59f2c20" containerName="maas-api" Aug 01 01:08:57.468320 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468277 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1196a223-e4b8-4f34-b63e-a245f59f2c20" containerName="maas-api" Aug 01 01:08:57.468320 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468284 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6053929-2c77-4359-86c7-f50409b503af" containerName="payload-pre-processing" Aug 01 01:08:57.468320 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468289 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6053929-2c77-4359-86c7-f50409b503af" containerName="payload-pre-processing" Aug 01 01:08:57.468634 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468346 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c468f6-2855-45c0-980b-135a0db82d04" containerName="istio-proxy" Aug 01 01:08:57.468634 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468355 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6053929-2c77-4359-86c7-f50409b503af" containerName="payload-pre-processing" Aug 01 01:08:57.468634 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468363 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1196a223-e4b8-4f34-b63e-a245f59f2c20" containerName="maas-api" Aug 01 01:08:57.468634 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.468368 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c6ed20-3267-452c-bacc-0b559502e88a" containerName="payload-processing" Aug 01 01:08:57.471490 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.471470 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.473906 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.473886 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-47a30d-openshift-default-dockercfg-6vf4s\"" Aug 01 01:08:57.482342 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.482314 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn"] Aug 01 01:08:57.654812 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.654769 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95b6023-80b6-4fef-ae57-5529cdb8b650-istiod-ca-cert\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.654812 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.654812 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-workload-socket\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.655051 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.654860 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-envoy\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.655051 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.654908 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-workload-certs\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.655051 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.654933 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-podinfo\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.655051 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.654969 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-credential-socket\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.655051 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.655006 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-data\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.655051 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.655045 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-token\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.655318 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.655130 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6sl8\" (UniqueName: \"kubernetes.io/projected/a95b6023-80b6-4fef-ae57-5529cdb8b650-kube-api-access-b6sl8\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756129 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756027 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95b6023-80b6-4fef-ae57-5529cdb8b650-istiod-ca-cert\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756129 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756101 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-workload-socket\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756357 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-envoy\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756357 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756169 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-workload-certs\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756357 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756199 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-podinfo\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756357 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756236 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-credential-socket\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756357 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756271 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-data\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756357 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756330 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-token\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756640 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756377 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6sl8\" (UniqueName: \"kubernetes.io/projected/a95b6023-80b6-4fef-ae57-5529cdb8b650-kube-api-access-b6sl8\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756640 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756601 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-workload-certs\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756754 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756664 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-credential-socket\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756754 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756713 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-workload-socket\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756897 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756876 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-data\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.756941 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.756896 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95b6023-80b6-4fef-ae57-5529cdb8b650-istiod-ca-cert\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.758535 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.758505 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-envoy\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.758765 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.758740 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-podinfo\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.764383 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.764357 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-token\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.764490 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.764476 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6sl8\" (UniqueName: \"kubernetes.io/projected/a95b6023-80b6-4fef-ae57-5529cdb8b650-kube-api-access-b6sl8\") pod \"e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.783432 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.783397 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:57.926736 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.926705 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn"] Aug 01 01:08:57.928643 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:08:57.928614 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95b6023_80b6_4fef_ae57_5529cdb8b650.slice/crio-9b204a156610459bab11b6076f5a29ff187f472aae2b0c644bcb86ccb8b4268d WatchSource:0}: Error finding container 9b204a156610459bab11b6076f5a29ff187f472aae2b0c644bcb86ccb8b4268d: Status 404 returned error can't find the container with id 9b204a156610459bab11b6076f5a29ff187f472aae2b0c644bcb86ccb8b4268d Aug 01 01:08:57.931184 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.931144 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:08:57.931268 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.931212 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:08:57.931268 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:57.931243 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:08:58.052800 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:58.052763 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" event={"ID":"a95b6023-80b6-4fef-ae57-5529cdb8b650","Type":"ContainerStarted","Data":"952381e1eb356017e56f982c781d1771ea7c164756c5870cdc3bd91ab8a34f0e"} Aug 01 01:08:58.053080 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:58.052820 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" event={"ID":"a95b6023-80b6-4fef-ae57-5529cdb8b650","Type":"ContainerStarted","Data":"9b204a156610459bab11b6076f5a29ff187f472aae2b0c644bcb86ccb8b4268d"} Aug 01 01:08:58.072155 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:58.072104 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" podStartSLOduration=1.072087931 podStartE2EDuration="1.072087931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:08:58.071478127 +0000 UTC m=+2426.860200023" watchObservedRunningTime="2026-08-01 01:08:58.072087931 +0000 UTC m=+2426.860809756" Aug 01 01:08:58.784308 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:58.784272 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:08:59.789689 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:08:59.789644 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:09:00.063427 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:00.063395 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:09:00.064429 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:00.064408 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:09:01.045874 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.045817 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv"] Aug 01 01:09:01.046489 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.046466 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c468f6-2855-45c0-980b-135a0db82d04" containerName="istio-proxy" Aug 01 01:09:01.049560 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.049542 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:01.052191 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.052173 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:09:01.052283 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.052223 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 01 01:09:01.053249 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.053230 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-47a30d-serving-cert\"" Aug 01 01:09:01.053305 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.053260 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gf2hx\"" Aug 01 01:09:01.062741 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.062698 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv"] Aug 01 01:09:01.188853 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.188815 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxl6\" (UniqueName: \"kubernetes.io/projected/8e8251cb-63ca-41d7-a027-e3d130b4731f-kube-api-access-6zxl6\") pod \"maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv\" (UID: \"8e8251cb-63ca-41d7-a027-e3d130b4731f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:01.189037 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.188966 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8251cb-63ca-41d7-a027-e3d130b4731f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv\" (UID: \"8e8251cb-63ca-41d7-a027-e3d130b4731f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:01.189189 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.189169 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8251cb-63ca-41d7-a027-e3d130b4731f-maas-api-tls\") pod \"maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv\" (UID: \"8e8251cb-63ca-41d7-a027-e3d130b4731f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:01.289856 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.289818 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8251cb-63ca-41d7-a027-e3d130b4731f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv\" (UID: \"8e8251cb-63ca-41d7-a027-e3d130b4731f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:01.290039 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.289880 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8251cb-63ca-41d7-a027-e3d130b4731f-maas-api-tls\") pod \"maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv\" (UID: \"8e8251cb-63ca-41d7-a027-e3d130b4731f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:01.290039 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.289943 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxl6\" (UniqueName: \"kubernetes.io/projected/8e8251cb-63ca-41d7-a027-e3d130b4731f-kube-api-access-6zxl6\") pod \"maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv\" (UID: \"8e8251cb-63ca-41d7-a027-e3d130b4731f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:01.290039 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:09:01.289977 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-serving-cert: secret "maas-api-e2e-inf-a-47a30d-serving-cert" not found Aug 01 01:09:01.290192 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:09:01.290046 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e8251cb-63ca-41d7-a027-e3d130b4731f-maas-api-tls podName:8e8251cb-63ca-41d7-a027-e3d130b4731f nodeName:}" failed. No retries permitted until 2026-08-01 01:09:01.790030438 +0000 UTC m=+2430.578752242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e8251cb-63ca-41d7-a027-e3d130b4731f-maas-api-tls") pod "maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" (UID: "8e8251cb-63ca-41d7-a027-e3d130b4731f") : secret "maas-api-e2e-inf-a-47a30d-serving-cert" not found Aug 01 01:09:01.290482 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.290462 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8251cb-63ca-41d7-a027-e3d130b4731f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv\" (UID: \"8e8251cb-63ca-41d7-a027-e3d130b4731f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:01.305978 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.305938 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxl6\" (UniqueName: \"kubernetes.io/projected/8e8251cb-63ca-41d7-a027-e3d130b4731f-kube-api-access-6zxl6\") pod \"maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv\" (UID: \"8e8251cb-63ca-41d7-a027-e3d130b4731f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:01.793714 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.793621 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8251cb-63ca-41d7-a027-e3d130b4731f-maas-api-tls\") pod \"maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv\" (UID: \"8e8251cb-63ca-41d7-a027-e3d130b4731f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:01.797675 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.797633 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8251cb-63ca-41d7-a027-e3d130b4731f-maas-api-tls\") pod \"maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv\" (UID: \"8e8251cb-63ca-41d7-a027-e3d130b4731f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:01.960959 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:01.960921 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:02.104563 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.104508 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv"] Aug 01 01:09:02.106812 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:09:02.106777 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8251cb_63ca_41d7_a027_e3d130b4731f.slice/crio-bbc69c1a4fb430e362c2e0e8c035bc9814d0e7d452d53b0b39700431862e4bdb WatchSource:0}: Error finding container bbc69c1a4fb430e362c2e0e8c035bc9814d0e7d452d53b0b39700431862e4bdb: Status 404 returned error can't find the container with id bbc69c1a4fb430e362c2e0e8c035bc9814d0e7d452d53b0b39700431862e4bdb Aug 01 01:09:02.130695 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.130661 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w"] Aug 01 01:09:02.135653 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.135632 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" Aug 01 01:09:02.138351 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.138330 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-47a30d\"" Aug 01 01:09:02.138487 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.138471 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-47a30d-dockercfg-jnckh\"" Aug 01 01:09:02.151775 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.151746 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w"] Aug 01 01:09:02.196881 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.196839 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77078e0d-ddc4-4200-a1fe-d81f8531bbcc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w\" (UID: \"77078e0d-ddc4-4200-a1fe-d81f8531bbcc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" Aug 01 01:09:02.197089 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.196898 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp7mr\" (UniqueName: \"kubernetes.io/projected/77078e0d-ddc4-4200-a1fe-d81f8531bbcc-kube-api-access-wp7mr\") pod \"payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w\" (UID: \"77078e0d-ddc4-4200-a1fe-d81f8531bbcc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" Aug 01 01:09:02.298206 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.298102 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77078e0d-ddc4-4200-a1fe-d81f8531bbcc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w\" (UID: \"77078e0d-ddc4-4200-a1fe-d81f8531bbcc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" Aug 01 01:09:02.298206 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.298174 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp7mr\" (UniqueName: \"kubernetes.io/projected/77078e0d-ddc4-4200-a1fe-d81f8531bbcc-kube-api-access-wp7mr\") pod \"payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w\" (UID: \"77078e0d-ddc4-4200-a1fe-d81f8531bbcc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" Aug 01 01:09:02.298838 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.298812 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77078e0d-ddc4-4200-a1fe-d81f8531bbcc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w\" (UID: \"77078e0d-ddc4-4200-a1fe-d81f8531bbcc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" Aug 01 01:09:02.306466 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.306438 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp7mr\" (UniqueName: \"kubernetes.io/projected/77078e0d-ddc4-4200-a1fe-d81f8531bbcc-kube-api-access-wp7mr\") pod \"payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w\" (UID: \"77078e0d-ddc4-4200-a1fe-d81f8531bbcc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" Aug 01 01:09:02.447761 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.447721 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" Aug 01 01:09:02.591860 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:02.591724 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w"] Aug 01 01:09:02.594559 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:09:02.594519 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77078e0d_ddc4_4200_a1fe_d81f8531bbcc.slice/crio-5cac097c18312229f9b8f8cd480670b19c0605f3b2cf499c82f0252d704c28b7 WatchSource:0}: Error finding container 5cac097c18312229f9b8f8cd480670b19c0605f3b2cf499c82f0252d704c28b7: Status 404 returned error can't find the container with id 5cac097c18312229f9b8f8cd480670b19c0605f3b2cf499c82f0252d704c28b7 Aug 01 01:09:03.084442 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:03.084387 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" event={"ID":"8e8251cb-63ca-41d7-a027-e3d130b4731f","Type":"ContainerStarted","Data":"23aee0b6bceededb2dbdab56075fcf64a10e988d9cdcebe6d27f33208239a61e"} Aug 01 01:09:03.084731 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:03.084599 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:03.084731 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:03.084641 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" event={"ID":"8e8251cb-63ca-41d7-a027-e3d130b4731f","Type":"ContainerStarted","Data":"bbc69c1a4fb430e362c2e0e8c035bc9814d0e7d452d53b0b39700431862e4bdb"} Aug 01 01:09:03.085923 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:03.085898 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" event={"ID":"77078e0d-ddc4-4200-a1fe-d81f8531bbcc","Type":"ContainerStarted","Data":"0e4f9ac540e510e68eb89ba7629e546e71e3f8e690daad3d1a5c7075e83887df"} Aug 01 01:09:03.086080 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:03.085938 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" event={"ID":"77078e0d-ddc4-4200-a1fe-d81f8531bbcc","Type":"ContainerStarted","Data":"5cac097c18312229f9b8f8cd480670b19c0605f3b2cf499c82f0252d704c28b7"} Aug 01 01:09:03.086080 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:03.086026 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" Aug 01 01:09:03.102863 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:03.102805 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" podStartSLOduration=1.6813842110000001 podStartE2EDuration="2.102774568s" totalImagesPullingTime="421.390357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:09:03.101148195 +0000 UTC m=+2431.889870047" watchObservedRunningTime="2026-08-01 01:09:03.102774568 +0000 UTC m=+2431.891496654" Aug 01 01:09:03.119642 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:03.119378 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" podStartSLOduration=1.119356438 podStartE2EDuration="1.119356438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:09:03.11877486 +0000 UTC m=+2431.907496687" watchObservedRunningTime="2026-08-01 01:09:03.119356438 +0000 UTC m=+2431.908078264" Aug 01 01:09:09.097423 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:09.097395 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:09:14.093467 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:14.093437 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" Aug 01 01:09:14.155728 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:14.155696 2563 generic.go:356] "Generic (PLEG): container finished" podID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerID="952381e1eb356017e56f982c781d1771ea7c164756c5870cdc3bd91ab8a34f0e" exitCode=0 Aug 01 01:09:14.155891 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:14.155752 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" event={"ID":"a95b6023-80b6-4fef-ae57-5529cdb8b650","Type":"ContainerDied","Data":"952381e1eb356017e56f982c781d1771ea7c164756c5870cdc3bd91ab8a34f0e"} Aug 01 01:09:14.156222 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:14.156207 2563 scope.go:117] "RemoveContainer" containerID="952381e1eb356017e56f982c781d1771ea7c164756c5870cdc3bd91ab8a34f0e" Aug 01 01:09:14.157421 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:14.157387 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:09:14.157537 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:14.157489 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:09:14.157581 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:14.157534 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:09:15.063913 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:15.063877 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:09:15.162028 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:15.161990 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" event={"ID":"a95b6023-80b6-4fef-ae57-5529cdb8b650","Type":"ContainerStarted","Data":"605e9b96fd8ffc7b903a2f6cdd174da30ea6ab5f0b249e56f1ae07cd5962d5c4"} Aug 01 01:09:15.783904 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:15.783865 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:09:15.784972 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:15.784947 2563 patch_prober.go:28] interesting pod/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Aug 01 01:09:15.785112 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:15.784994 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Aug 01 01:09:16.784197 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:16.784154 2563 patch_prober.go:28] interesting pod/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Aug 01 01:09:16.784664 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:16.784215 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Aug 01 01:09:17.783852 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:17.783816 2563 patch_prober.go:28] interesting pod/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Aug 01 01:09:17.784021 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:17.783868 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Aug 01 01:09:18.784086 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:18.784037 2563 patch_prober.go:28] interesting pod/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Aug 01 01:09:18.784460 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:18.784104 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Aug 01 01:09:19.784707 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:19.784669 2563 patch_prober.go:28] interesting pod/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Aug 01 01:09:19.785093 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:19.784718 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Aug 01 01:09:20.787344 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:20.787317 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:09:20.787705 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:20.787576 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:09:20.788151 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:20.788133 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:09:25.530572 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.530531 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw"] Aug 01 01:09:25.534791 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.534760 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.537340 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.537306 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-061b54-openshift-default-dockercfg-6gzn8\"" Aug 01 01:09:25.543857 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.543833 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw"] Aug 01 01:09:25.632182 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.632138 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-workload-certs\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.632390 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.632201 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istiod-ca-cert\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.632390 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.632236 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-token\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.632390 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.632276 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-workload-socket\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.632390 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.632372 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-data\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.632630 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.632413 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-envoy\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.632630 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.632444 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjjq6\" (UniqueName: \"kubernetes.io/projected/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-kube-api-access-tjjq6\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.632630 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.632477 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-podinfo\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.632630 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.632514 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-credential-socket\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.733879 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.733838 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-data\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.733879 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.733882 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-envoy\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.734175 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.733906 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjjq6\" (UniqueName: \"kubernetes.io/projected/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-kube-api-access-tjjq6\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.734175 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.733931 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-podinfo\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.734175 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.734028 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-credential-socket\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.734175 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.734157 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-workload-certs\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.734442 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.734198 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istiod-ca-cert\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.734442 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.734233 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-token\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.734442 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.734261 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-workload-socket\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.734442 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.734425 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-credential-socket\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.734722 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.734484 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-workload-certs\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.734722 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.734575 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-workload-socket\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.734722 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.734620 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-data\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.734899 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.734817 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istiod-ca-cert\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.736153 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.736133 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-envoy\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.736374 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.736356 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-podinfo\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.747635 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.747602 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-token\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.747736 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.747670 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjjq6\" (UniqueName: \"kubernetes.io/projected/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-kube-api-access-tjjq6\") pod \"e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.845854 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.845820 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:25.981101 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.981046 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw"] Aug 01 01:09:25.983213 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:09:25.983186 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f0d8cc_6d41_4ba3_8895_a40cc97a1096.slice/crio-f191912d8319a1e7957c48d430c4490e04300730640356ca3da2e40c87b98cc3 WatchSource:0}: Error finding container f191912d8319a1e7957c48d430c4490e04300730640356ca3da2e40c87b98cc3: Status 404 returned error can't find the container with id f191912d8319a1e7957c48d430c4490e04300730640356ca3da2e40c87b98cc3 Aug 01 01:09:25.985241 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.985209 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:09:25.985325 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.985271 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:09:25.985325 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:25.985300 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:09:26.217552 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:26.217466 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" event={"ID":"33f0d8cc-6d41-4ba3-8895-a40cc97a1096","Type":"ContainerStarted","Data":"3cade66f9648cdd04337266ff210e11bdd4b0f85c0cda14393ee5c7f2cd53af5"} Aug 01 01:09:26.217552 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:26.217518 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" event={"ID":"33f0d8cc-6d41-4ba3-8895-a40cc97a1096","Type":"ContainerStarted","Data":"f191912d8319a1e7957c48d430c4490e04300730640356ca3da2e40c87b98cc3"} Aug 01 01:09:26.238560 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:26.238494 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" podStartSLOduration=1.2384723420000001 podStartE2EDuration="1.238472342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:09:26.235624167 +0000 UTC m=+2455.024346006" watchObservedRunningTime="2026-08-01 01:09:26.238472342 +0000 UTC m=+2455.027194172" Aug 01 01:09:26.846958 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:26.846924 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:27.851717 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:27.851685 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:28.228121 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:28.228039 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:28.229109 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:28.229088 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:29.960048 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:29.960018 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg"] Aug 01 01:09:29.963933 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:29.963910 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" Aug 01 01:09:29.966548 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:29.966523 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-061b54\"" Aug 01 01:09:29.966656 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:29.966589 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-061b54-dockercfg-v8qvl\"" Aug 01 01:09:29.974626 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:29.974603 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg"] Aug 01 01:09:30.074077 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:30.074016 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpbfk\" (UniqueName: \"kubernetes.io/projected/ea3872ed-c8dc-4816-826b-401558d61b47-kube-api-access-zpbfk\") pod \"payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg\" (UID: \"ea3872ed-c8dc-4816-826b-401558d61b47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" Aug 01 01:09:30.074249 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:30.074093 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3872ed-c8dc-4816-826b-401558d61b47-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg\" (UID: \"ea3872ed-c8dc-4816-826b-401558d61b47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" Aug 01 01:09:30.175491 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:30.175453 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbfk\" (UniqueName: \"kubernetes.io/projected/ea3872ed-c8dc-4816-826b-401558d61b47-kube-api-access-zpbfk\") pod \"payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg\" (UID: \"ea3872ed-c8dc-4816-826b-401558d61b47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" Aug 01 01:09:30.175491 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:30.175494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3872ed-c8dc-4816-826b-401558d61b47-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg\" (UID: \"ea3872ed-c8dc-4816-826b-401558d61b47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" Aug 01 01:09:30.176195 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:30.176174 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3872ed-c8dc-4816-826b-401558d61b47-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg\" (UID: \"ea3872ed-c8dc-4816-826b-401558d61b47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" Aug 01 01:09:30.182730 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:30.182707 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbfk\" (UniqueName: \"kubernetes.io/projected/ea3872ed-c8dc-4816-826b-401558d61b47-kube-api-access-zpbfk\") pod \"payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg\" (UID: \"ea3872ed-c8dc-4816-826b-401558d61b47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" Aug 01 01:09:30.274849 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:30.274769 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" Aug 01 01:09:30.410653 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:30.410490 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg"] Aug 01 01:09:30.413270 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:09:30.413238 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3872ed_c8dc_4816_826b_401558d61b47.slice/crio-7486b02f0ad82dbc79825421c330b46eeeda7e064cc589cd7c32a731c46ae003 WatchSource:0}: Error finding container 7486b02f0ad82dbc79825421c330b46eeeda7e064cc589cd7c32a731c46ae003: Status 404 returned error can't find the container with id 7486b02f0ad82dbc79825421c330b46eeeda7e064cc589cd7c32a731c46ae003 Aug 01 01:09:31.243394 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:31.243360 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" event={"ID":"ea3872ed-c8dc-4816-826b-401558d61b47","Type":"ContainerStarted","Data":"ace3a1add2bcdb5ca157dbdec8c1ce48f6e1a521cca4635e691d1e3a872eff5d"} Aug 01 01:09:31.243784 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:31.243408 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" event={"ID":"ea3872ed-c8dc-4816-826b-401558d61b47","Type":"ContainerStarted","Data":"7486b02f0ad82dbc79825421c330b46eeeda7e064cc589cd7c32a731c46ae003"} Aug 01 01:09:31.243784 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:31.243503 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" Aug 01 01:09:31.260327 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:31.260282 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" podStartSLOduration=2.260266481 podStartE2EDuration="2.260266481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:09:31.258851182 +0000 UTC m=+2460.047572987" watchObservedRunningTime="2026-08-01 01:09:31.260266481 +0000 UTC m=+2460.048988307" Aug 01 01:09:42.250385 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:42.250352 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" Aug 01 01:09:42.297091 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:42.297043 2563 generic.go:356] "Generic (PLEG): container finished" podID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerID="3cade66f9648cdd04337266ff210e11bdd4b0f85c0cda14393ee5c7f2cd53af5" exitCode=0 Aug 01 01:09:42.297252 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:42.297123 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" event={"ID":"33f0d8cc-6d41-4ba3-8895-a40cc97a1096","Type":"ContainerDied","Data":"3cade66f9648cdd04337266ff210e11bdd4b0f85c0cda14393ee5c7f2cd53af5"} Aug 01 01:09:42.297613 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:42.297596 2563 scope.go:117] "RemoveContainer" containerID="3cade66f9648cdd04337266ff210e11bdd4b0f85c0cda14393ee5c7f2cd53af5" Aug 01 01:09:42.298733 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:42.298706 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:09:42.298801 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:42.298768 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:09:42.298837 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:42.298799 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:09:43.228530 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:43.228482 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:43.303589 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:43.303558 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" event={"ID":"33f0d8cc-6d41-4ba3-8895-a40cc97a1096","Type":"ContainerStarted","Data":"2126dd57e3203632f6d35ab872f966ea91efac3676434fd9a7252ea55fb54245"} Aug 01 01:09:43.846301 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:43.846270 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:43.847199 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:43.847178 2563 patch_prober.go:28] interesting pod/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Aug 01 01:09:43.847309 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:43.847215 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" podUID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Aug 01 01:09:44.846763 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:44.846732 2563 patch_prober.go:28] interesting pod/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Aug 01 01:09:44.847183 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:44.846792 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" podUID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Aug 01 01:09:45.846890 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:45.846849 2563 patch_prober.go:28] interesting pod/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Aug 01 01:09:45.847384 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:45.846917 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" podUID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Aug 01 01:09:46.846273 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:46.846234 2563 patch_prober.go:28] interesting pod/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Aug 01 01:09:46.846456 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:46.846292 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" podUID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Aug 01 01:09:47.846972 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:47.846930 2563 patch_prober.go:28] interesting pod/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Aug 01 01:09:47.847376 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:47.847001 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" podUID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Aug 01 01:09:48.850270 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:48.850244 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:48.850634 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:48.850488 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:09:48.851218 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:09:48.851201 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:11:54.972472 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:54.972373 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv"] Aug 01 01:11:54.972917 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:54.972772 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" podUID="8e8251cb-63ca-41d7-a027-e3d130b4731f" containerName="maas-api" containerID="cri-o://23aee0b6bceededb2dbdab56075fcf64a10e988d9cdcebe6d27f33208239a61e" gracePeriod=30 Aug 01 01:11:55.026264 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.026232 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w"] Aug 01 01:11:55.026499 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.026475 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" podUID="77078e0d-ddc4-4200-a1fe-d81f8531bbcc" containerName="payload-processing" containerID="cri-o://0e4f9ac540e510e68eb89ba7629e546e71e3f8e690daad3d1a5c7075e83887df" gracePeriod=30 Aug 01 01:11:55.281434 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.281407 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:11:55.324127 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.324100 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" Aug 01 01:11:55.335721 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.335697 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8251cb-63ca-41d7-a027-e3d130b4731f-openshift-service-ca\") pod \"8e8251cb-63ca-41d7-a027-e3d130b4731f\" (UID: \"8e8251cb-63ca-41d7-a027-e3d130b4731f\") " Aug 01 01:11:55.335830 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.335728 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zxl6\" (UniqueName: \"kubernetes.io/projected/8e8251cb-63ca-41d7-a027-e3d130b4731f-kube-api-access-6zxl6\") pod \"8e8251cb-63ca-41d7-a027-e3d130b4731f\" (UID: \"8e8251cb-63ca-41d7-a027-e3d130b4731f\") " Aug 01 01:11:55.335830 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.335745 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8251cb-63ca-41d7-a027-e3d130b4731f-maas-api-tls\") pod \"8e8251cb-63ca-41d7-a027-e3d130b4731f\" (UID: \"8e8251cb-63ca-41d7-a027-e3d130b4731f\") " Aug 01 01:11:55.336023 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.336001 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8251cb-63ca-41d7-a027-e3d130b4731f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e8251cb-63ca-41d7-a027-e3d130b4731f" (UID: "8e8251cb-63ca-41d7-a027-e3d130b4731f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:11:55.336137 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.336116 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8251cb-63ca-41d7-a027-e3d130b4731f-openshift-service-ca\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:11:55.338342 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.338317 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8251cb-63ca-41d7-a027-e3d130b4731f-kube-api-access-6zxl6" (OuterVolumeSpecName: "kube-api-access-6zxl6") pod "8e8251cb-63ca-41d7-a027-e3d130b4731f" (UID: "8e8251cb-63ca-41d7-a027-e3d130b4731f"). InnerVolumeSpecName "kube-api-access-6zxl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:11:55.338445 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.338352 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8251cb-63ca-41d7-a027-e3d130b4731f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e8251cb-63ca-41d7-a027-e3d130b4731f" (UID: "8e8251cb-63ca-41d7-a027-e3d130b4731f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:11:55.436633 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.436601 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77078e0d-ddc4-4200-a1fe-d81f8531bbcc-plugins-config-volume\") pod \"77078e0d-ddc4-4200-a1fe-d81f8531bbcc\" (UID: \"77078e0d-ddc4-4200-a1fe-d81f8531bbcc\") " Aug 01 01:11:55.436633 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.436639 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp7mr\" (UniqueName: \"kubernetes.io/projected/77078e0d-ddc4-4200-a1fe-d81f8531bbcc-kube-api-access-wp7mr\") pod \"77078e0d-ddc4-4200-a1fe-d81f8531bbcc\" (UID: \"77078e0d-ddc4-4200-a1fe-d81f8531bbcc\") " Aug 01 01:11:55.436865 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.436853 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxl6\" (UniqueName: \"kubernetes.io/projected/8e8251cb-63ca-41d7-a027-e3d130b4731f-kube-api-access-6zxl6\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:11:55.436907 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.436867 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8251cb-63ca-41d7-a027-e3d130b4731f-maas-api-tls\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:11:55.437093 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.437039 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77078e0d-ddc4-4200-a1fe-d81f8531bbcc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77078e0d-ddc4-4200-a1fe-d81f8531bbcc" (UID: "77078e0d-ddc4-4200-a1fe-d81f8531bbcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:11:55.438680 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.438654 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77078e0d-ddc4-4200-a1fe-d81f8531bbcc-kube-api-access-wp7mr" (OuterVolumeSpecName: "kube-api-access-wp7mr") pod "77078e0d-ddc4-4200-a1fe-d81f8531bbcc" (UID: "77078e0d-ddc4-4200-a1fe-d81f8531bbcc"). InnerVolumeSpecName "kube-api-access-wp7mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:11:55.544114 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.538582 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77078e0d-ddc4-4200-a1fe-d81f8531bbcc-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:11:55.544114 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.538618 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp7mr\" (UniqueName: \"kubernetes.io/projected/77078e0d-ddc4-4200-a1fe-d81f8531bbcc-kube-api-access-wp7mr\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:11:55.954400 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.954370 2563 generic.go:356] "Generic (PLEG): container finished" podID="8e8251cb-63ca-41d7-a027-e3d130b4731f" containerID="23aee0b6bceededb2dbdab56075fcf64a10e988d9cdcebe6d27f33208239a61e" exitCode=0 Aug 01 01:11:55.954600 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.954424 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" Aug 01 01:11:55.954600 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.954470 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" event={"ID":"8e8251cb-63ca-41d7-a027-e3d130b4731f","Type":"ContainerDied","Data":"23aee0b6bceededb2dbdab56075fcf64a10e988d9cdcebe6d27f33208239a61e"} Aug 01 01:11:55.954600 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.954522 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv" event={"ID":"8e8251cb-63ca-41d7-a027-e3d130b4731f","Type":"ContainerDied","Data":"bbc69c1a4fb430e362c2e0e8c035bc9814d0e7d452d53b0b39700431862e4bdb"} Aug 01 01:11:55.954600 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.954544 2563 scope.go:117] "RemoveContainer" containerID="23aee0b6bceededb2dbdab56075fcf64a10e988d9cdcebe6d27f33208239a61e" Aug 01 01:11:55.955954 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.955934 2563 generic.go:356] "Generic (PLEG): container finished" podID="77078e0d-ddc4-4200-a1fe-d81f8531bbcc" containerID="0e4f9ac540e510e68eb89ba7629e546e71e3f8e690daad3d1a5c7075e83887df" exitCode=0 Aug 01 01:11:55.956106 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.955983 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" Aug 01 01:11:55.956106 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.955995 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" event={"ID":"77078e0d-ddc4-4200-a1fe-d81f8531bbcc","Type":"ContainerDied","Data":"0e4f9ac540e510e68eb89ba7629e546e71e3f8e690daad3d1a5c7075e83887df"} Aug 01 01:11:55.956226 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.956115 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w" event={"ID":"77078e0d-ddc4-4200-a1fe-d81f8531bbcc","Type":"ContainerDied","Data":"5cac097c18312229f9b8f8cd480670b19c0605f3b2cf499c82f0252d704c28b7"} Aug 01 01:11:55.963237 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.963219 2563 scope.go:117] "RemoveContainer" containerID="23aee0b6bceededb2dbdab56075fcf64a10e988d9cdcebe6d27f33208239a61e" Aug 01 01:11:55.963470 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:11:55.963453 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23aee0b6bceededb2dbdab56075fcf64a10e988d9cdcebe6d27f33208239a61e\": container with ID starting with 23aee0b6bceededb2dbdab56075fcf64a10e988d9cdcebe6d27f33208239a61e not found: ID does not exist" containerID="23aee0b6bceededb2dbdab56075fcf64a10e988d9cdcebe6d27f33208239a61e" Aug 01 01:11:55.963528 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.963476 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23aee0b6bceededb2dbdab56075fcf64a10e988d9cdcebe6d27f33208239a61e"} err="failed to get container status \"23aee0b6bceededb2dbdab56075fcf64a10e988d9cdcebe6d27f33208239a61e\": rpc error: code = NotFound desc = could not find container \"23aee0b6bceededb2dbdab56075fcf64a10e988d9cdcebe6d27f33208239a61e\": container with ID starting with 23aee0b6bceededb2dbdab56075fcf64a10e988d9cdcebe6d27f33208239a61e not found: ID does not exist" Aug 01 01:11:55.963528 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.963503 2563 scope.go:117] "RemoveContainer" containerID="0e4f9ac540e510e68eb89ba7629e546e71e3f8e690daad3d1a5c7075e83887df" Aug 01 01:11:55.971439 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.971422 2563 scope.go:117] "RemoveContainer" containerID="0e4f9ac540e510e68eb89ba7629e546e71e3f8e690daad3d1a5c7075e83887df" Aug 01 01:11:55.971707 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:11:55.971691 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e4f9ac540e510e68eb89ba7629e546e71e3f8e690daad3d1a5c7075e83887df\": container with ID starting with 0e4f9ac540e510e68eb89ba7629e546e71e3f8e690daad3d1a5c7075e83887df not found: ID does not exist" containerID="0e4f9ac540e510e68eb89ba7629e546e71e3f8e690daad3d1a5c7075e83887df" Aug 01 01:11:55.971768 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.971713 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e4f9ac540e510e68eb89ba7629e546e71e3f8e690daad3d1a5c7075e83887df"} err="failed to get container status \"0e4f9ac540e510e68eb89ba7629e546e71e3f8e690daad3d1a5c7075e83887df\": rpc error: code = NotFound desc = could not find container \"0e4f9ac540e510e68eb89ba7629e546e71e3f8e690daad3d1a5c7075e83887df\": container with ID starting with 0e4f9ac540e510e68eb89ba7629e546e71e3f8e690daad3d1a5c7075e83887df not found: ID does not exist" Aug 01 01:11:55.972766 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.972749 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w"] Aug 01 01:11:55.975711 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.975691 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47a30d-7cf87989bd-cq78w"] Aug 01 01:11:55.986021 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.985988 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv"] Aug 01 01:11:55.987816 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:55.987795 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47a30d-85849fbfc8-dsctv"] Aug 01 01:11:57.728368 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:57.728337 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77078e0d-ddc4-4200-a1fe-d81f8531bbcc" path="/var/lib/kubelet/pods/77078e0d-ddc4-4200-a1fe-d81f8531bbcc/volumes" Aug 01 01:11:57.728750 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:11:57.728736 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e8251cb-63ca-41d7-a027-e3d130b4731f" path="/var/lib/kubelet/pods/8e8251cb-63ca-41d7-a027-e3d130b4731f/volumes" Aug 01 01:12:01.831760 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:01.831722 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn"] Aug 01 01:12:01.832213 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:01.831963 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerName="istio-proxy" containerID="cri-o://605e9b96fd8ffc7b903a2f6cdd174da30ea6ab5f0b249e56f1ae07cd5962d5c4" gracePeriod=30 Aug 01 01:12:05.790783 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:05.790740 2563 patch_prober.go:28] interesting pod/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 01 01:12:05.791187 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:05.790799 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 01 01:12:07.024171 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.024117 2563 generic.go:356] "Generic (PLEG): container finished" podID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerID="605e9b96fd8ffc7b903a2f6cdd174da30ea6ab5f0b249e56f1ae07cd5962d5c4" exitCode=0 Aug 01 01:12:07.024673 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.024179 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" event={"ID":"a95b6023-80b6-4fef-ae57-5529cdb8b650","Type":"ContainerDied","Data":"605e9b96fd8ffc7b903a2f6cdd174da30ea6ab5f0b249e56f1ae07cd5962d5c4"} Aug 01 01:12:07.024673 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.024262 2563 scope.go:117] "RemoveContainer" containerID="952381e1eb356017e56f982c781d1771ea7c164756c5870cdc3bd91ab8a34f0e" Aug 01 01:12:07.084597 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.084569 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:12:07.145248 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.145165 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-credential-socket\") pod \"a95b6023-80b6-4fef-ae57-5529cdb8b650\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " Aug 01 01:12:07.145248 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.145208 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-token\") pod \"a95b6023-80b6-4fef-ae57-5529cdb8b650\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " Aug 01 01:12:07.145440 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.145254 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6sl8\" (UniqueName: \"kubernetes.io/projected/a95b6023-80b6-4fef-ae57-5529cdb8b650-kube-api-access-b6sl8\") pod \"a95b6023-80b6-4fef-ae57-5529cdb8b650\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " Aug 01 01:12:07.145440 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.145318 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-workload-certs\") pod \"a95b6023-80b6-4fef-ae57-5529cdb8b650\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " Aug 01 01:12:07.145554 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.145451 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-data\") pod \"a95b6023-80b6-4fef-ae57-5529cdb8b650\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " Aug 01 01:12:07.145554 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.145480 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a95b6023-80b6-4fef-ae57-5529cdb8b650" (UID: "a95b6023-80b6-4fef-ae57-5529cdb8b650"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:12:07.145554 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.145535 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-workload-socket\") pod \"a95b6023-80b6-4fef-ae57-5529cdb8b650\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " Aug 01 01:12:07.145705 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.145564 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-envoy\") pod \"a95b6023-80b6-4fef-ae57-5529cdb8b650\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " Aug 01 01:12:07.145705 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.145582 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a95b6023-80b6-4fef-ae57-5529cdb8b650" (UID: "a95b6023-80b6-4fef-ae57-5529cdb8b650"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:12:07.145705 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.145629 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95b6023-80b6-4fef-ae57-5529cdb8b650-istiod-ca-cert\") pod \"a95b6023-80b6-4fef-ae57-5529cdb8b650\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " Aug 01 01:12:07.145705 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.145698 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-podinfo\") pod \"a95b6023-80b6-4fef-ae57-5529cdb8b650\" (UID: \"a95b6023-80b6-4fef-ae57-5529cdb8b650\") " Aug 01 01:12:07.145901 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.145743 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-data" (OuterVolumeSpecName: "istio-data") pod "a95b6023-80b6-4fef-ae57-5529cdb8b650" (UID: "a95b6023-80b6-4fef-ae57-5529cdb8b650"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:12:07.145901 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.145775 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a95b6023-80b6-4fef-ae57-5529cdb8b650" (UID: "a95b6023-80b6-4fef-ae57-5529cdb8b650"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:12:07.146021 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.146003 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-workload-certs\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:07.146100 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.146027 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-data\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:07.146100 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.146040 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-workload-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:07.146100 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.146069 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-credential-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:07.146100 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.146082 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95b6023-80b6-4fef-ae57-5529cdb8b650-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a95b6023-80b6-4fef-ae57-5529cdb8b650" (UID: "a95b6023-80b6-4fef-ae57-5529cdb8b650"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:12:07.147549 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.147527 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-token" (OuterVolumeSpecName: "istio-token") pod "a95b6023-80b6-4fef-ae57-5529cdb8b650" (UID: "a95b6023-80b6-4fef-ae57-5529cdb8b650"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:12:07.147675 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.147557 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95b6023-80b6-4fef-ae57-5529cdb8b650-kube-api-access-b6sl8" (OuterVolumeSpecName: "kube-api-access-b6sl8") pod "a95b6023-80b6-4fef-ae57-5529cdb8b650" (UID: "a95b6023-80b6-4fef-ae57-5529cdb8b650"). InnerVolumeSpecName "kube-api-access-b6sl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:12:07.147941 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.147921 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a95b6023-80b6-4fef-ae57-5529cdb8b650" (UID: "a95b6023-80b6-4fef-ae57-5529cdb8b650"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:12:07.148007 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.147928 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a95b6023-80b6-4fef-ae57-5529cdb8b650" (UID: "a95b6023-80b6-4fef-ae57-5529cdb8b650"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 01 01:12:07.247095 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.247009 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-envoy\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:07.247095 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.247089 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95b6023-80b6-4fef-ae57-5529cdb8b650-istiod-ca-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:07.247095 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.247101 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-podinfo\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:07.247095 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.247109 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95b6023-80b6-4fef-ae57-5529cdb8b650-istio-token\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:07.247368 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:07.247118 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6sl8\" (UniqueName: \"kubernetes.io/projected/a95b6023-80b6-4fef-ae57-5529cdb8b650-kube-api-access-b6sl8\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:08.031211 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:08.031134 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" Aug 01 01:12:08.031632 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:08.031123 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn" event={"ID":"a95b6023-80b6-4fef-ae57-5529cdb8b650","Type":"ContainerDied","Data":"9b204a156610459bab11b6076f5a29ff187f472aae2b0c644bcb86ccb8b4268d"} Aug 01 01:12:08.031632 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:08.031267 2563 scope.go:117] "RemoveContainer" containerID="605e9b96fd8ffc7b903a2f6cdd174da30ea6ab5f0b249e56f1ae07cd5962d5c4" Aug 01 01:12:08.049800 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:08.049771 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn"] Aug 01 01:12:08.053443 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:08.053418 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-47a30d-openshift-default-56d9649ff6-wrcwn"] Aug 01 01:12:09.728023 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:09.727990 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" path="/var/lib/kubelet/pods/a95b6023-80b6-4fef-ae57-5529cdb8b650/volumes" Aug 01 01:12:12.668805 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:12.668772 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg"] Aug 01 01:12:12.669453 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:12.669113 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" podUID="ea3872ed-c8dc-4816-826b-401558d61b47" containerName="payload-pre-processing" containerID="cri-o://ace3a1add2bcdb5ca157dbdec8c1ce48f6e1a521cca4635e691d1e3a872eff5d" gracePeriod=30 Aug 01 01:12:12.930553 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:12.930489 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" Aug 01 01:12:13.003992 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.003957 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpbfk\" (UniqueName: \"kubernetes.io/projected/ea3872ed-c8dc-4816-826b-401558d61b47-kube-api-access-zpbfk\") pod \"ea3872ed-c8dc-4816-826b-401558d61b47\" (UID: \"ea3872ed-c8dc-4816-826b-401558d61b47\") " Aug 01 01:12:13.004175 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.004021 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3872ed-c8dc-4816-826b-401558d61b47-plugins-config-volume\") pod \"ea3872ed-c8dc-4816-826b-401558d61b47\" (UID: \"ea3872ed-c8dc-4816-826b-401558d61b47\") " Aug 01 01:12:13.004435 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.004410 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3872ed-c8dc-4816-826b-401558d61b47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea3872ed-c8dc-4816-826b-401558d61b47" (UID: "ea3872ed-c8dc-4816-826b-401558d61b47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:12:13.006030 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.006007 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3872ed-c8dc-4816-826b-401558d61b47-kube-api-access-zpbfk" (OuterVolumeSpecName: "kube-api-access-zpbfk") pod "ea3872ed-c8dc-4816-826b-401558d61b47" (UID: "ea3872ed-c8dc-4816-826b-401558d61b47"). InnerVolumeSpecName "kube-api-access-zpbfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:12:13.057212 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.057173 2563 generic.go:356] "Generic (PLEG): container finished" podID="ea3872ed-c8dc-4816-826b-401558d61b47" containerID="ace3a1add2bcdb5ca157dbdec8c1ce48f6e1a521cca4635e691d1e3a872eff5d" exitCode=0 Aug 01 01:12:13.057212 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.057213 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" Aug 01 01:12:13.057539 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.057258 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" event={"ID":"ea3872ed-c8dc-4816-826b-401558d61b47","Type":"ContainerDied","Data":"ace3a1add2bcdb5ca157dbdec8c1ce48f6e1a521cca4635e691d1e3a872eff5d"} Aug 01 01:12:13.057539 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.057316 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg" event={"ID":"ea3872ed-c8dc-4816-826b-401558d61b47","Type":"ContainerDied","Data":"7486b02f0ad82dbc79825421c330b46eeeda7e064cc589cd7c32a731c46ae003"} Aug 01 01:12:13.057539 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.057345 2563 scope.go:117] "RemoveContainer" containerID="ace3a1add2bcdb5ca157dbdec8c1ce48f6e1a521cca4635e691d1e3a872eff5d" Aug 01 01:12:13.066356 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.066341 2563 scope.go:117] "RemoveContainer" containerID="ace3a1add2bcdb5ca157dbdec8c1ce48f6e1a521cca4635e691d1e3a872eff5d" Aug 01 01:12:13.066600 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:12:13.066583 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace3a1add2bcdb5ca157dbdec8c1ce48f6e1a521cca4635e691d1e3a872eff5d\": container with ID starting with ace3a1add2bcdb5ca157dbdec8c1ce48f6e1a521cca4635e691d1e3a872eff5d not found: ID does not exist" containerID="ace3a1add2bcdb5ca157dbdec8c1ce48f6e1a521cca4635e691d1e3a872eff5d" Aug 01 01:12:13.066648 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.066607 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace3a1add2bcdb5ca157dbdec8c1ce48f6e1a521cca4635e691d1e3a872eff5d"} err="failed to get container status \"ace3a1add2bcdb5ca157dbdec8c1ce48f6e1a521cca4635e691d1e3a872eff5d\": rpc error: code = NotFound desc = could not find container \"ace3a1add2bcdb5ca157dbdec8c1ce48f6e1a521cca4635e691d1e3a872eff5d\": container with ID starting with ace3a1add2bcdb5ca157dbdec8c1ce48f6e1a521cca4635e691d1e3a872eff5d not found: ID does not exist" Aug 01 01:12:13.081547 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.081516 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg"] Aug 01 01:12:13.086533 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.086502 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-061b54-58d4654ddd-99ssg"] Aug 01 01:12:13.105100 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.105037 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpbfk\" (UniqueName: \"kubernetes.io/projected/ea3872ed-c8dc-4816-826b-401558d61b47-kube-api-access-zpbfk\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:13.105100 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.105098 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3872ed-c8dc-4816-826b-401558d61b47-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:13.729017 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:13.728984 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea3872ed-c8dc-4816-826b-401558d61b47" path="/var/lib/kubelet/pods/ea3872ed-c8dc-4816-826b-401558d61b47/volumes" Aug 01 01:12:19.266575 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:19.266542 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw"] Aug 01 01:12:19.266998 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:19.266743 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" podUID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerName="istio-proxy" containerID="cri-o://2126dd57e3203632f6d35ab872f966ea91efac3676434fd9a7252ea55fb54245" gracePeriod=30 Aug 01 01:12:20.108406 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.108377 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf"] Aug 01 01:12:20.108782 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.108767 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77078e0d-ddc4-4200-a1fe-d81f8531bbcc" containerName="payload-processing" Aug 01 01:12:20.108878 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.108786 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="77078e0d-ddc4-4200-a1fe-d81f8531bbcc" containerName="payload-processing" Aug 01 01:12:20.108878 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.108801 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerName="istio-proxy" Aug 01 01:12:20.108878 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.108807 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerName="istio-proxy" Aug 01 01:12:20.108878 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.108825 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e8251cb-63ca-41d7-a027-e3d130b4731f" containerName="maas-api" Aug 01 01:12:20.108878 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.108831 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e8251cb-63ca-41d7-a027-e3d130b4731f" containerName="maas-api" Aug 01 01:12:20.108878 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.108841 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea3872ed-c8dc-4816-826b-401558d61b47" containerName="payload-pre-processing" Aug 01 01:12:20.108878 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.108877 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea3872ed-c8dc-4816-826b-401558d61b47" containerName="payload-pre-processing" Aug 01 01:12:20.109278 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.108961 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e8251cb-63ca-41d7-a027-e3d130b4731f" containerName="maas-api" Aug 01 01:12:20.109278 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.109008 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="77078e0d-ddc4-4200-a1fe-d81f8531bbcc" containerName="payload-processing" Aug 01 01:12:20.109278 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.109019 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerName="istio-proxy" Aug 01 01:12:20.109278 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.109030 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea3872ed-c8dc-4816-826b-401558d61b47" containerName="payload-pre-processing" Aug 01 01:12:20.113355 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.113336 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.115899 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.115879 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3b6f87-openshift-default-dockercfg-x4nlt\"" Aug 01 01:12:20.125017 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.124995 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf"] Aug 01 01:12:20.279432 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.279398 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-envoy\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.279432 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.279438 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-workload-certs\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.279919 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.279501 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c4731a-1109-4eea-8300-2a38127ae1ff-istiod-ca-cert\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.279919 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.279579 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-data\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.279919 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.279617 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-podinfo\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.279919 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.279679 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-workload-socket\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.279919 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.279719 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-credential-socket\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.279919 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.279747 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntzdc\" (UniqueName: \"kubernetes.io/projected/f1c4731a-1109-4eea-8300-2a38127ae1ff-kube-api-access-ntzdc\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.279919 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.279768 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-token\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.380787 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.380712 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-workload-certs\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.380787 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.380759 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c4731a-1109-4eea-8300-2a38127ae1ff-istiod-ca-cert\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.380787 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.380784 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-data\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.381038 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.380806 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-podinfo\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.381038 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.380941 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-workload-socket\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.381038 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.380997 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-credential-socket\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.381038 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.381016 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntzdc\" (UniqueName: \"kubernetes.io/projected/f1c4731a-1109-4eea-8300-2a38127ae1ff-kube-api-access-ntzdc\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.381038 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.381033 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-token\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.381340 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.381116 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-envoy\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.381340 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.381195 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-workload-certs\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.381340 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.381215 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-data\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.381500 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.381352 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-workload-socket\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.381500 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.381424 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-credential-socket\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.381759 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.381740 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c4731a-1109-4eea-8300-2a38127ae1ff-istiod-ca-cert\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.383346 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.383327 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-podinfo\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.383430 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.383348 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-envoy\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.389151 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.389132 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntzdc\" (UniqueName: \"kubernetes.io/projected/f1c4731a-1109-4eea-8300-2a38127ae1ff-kube-api-access-ntzdc\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.389294 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.389278 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-token\") pod \"e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.427377 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.427352 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:20.571401 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.571378 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf"] Aug 01 01:12:20.573527 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:12:20.573501 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c4731a_1109_4eea_8300_2a38127ae1ff.slice/crio-8218eb50eb961ed2a71c7671b79880259eb6736645a5ffc370cd818aaf5901a2 WatchSource:0}: Error finding container 8218eb50eb961ed2a71c7671b79880259eb6736645a5ffc370cd818aaf5901a2: Status 404 returned error can't find the container with id 8218eb50eb961ed2a71c7671b79880259eb6736645a5ffc370cd818aaf5901a2 Aug 01 01:12:20.575591 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.575556 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:12:20.575680 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.575623 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:12:20.575680 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:20.575656 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:12:21.099302 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:21.099266 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" event={"ID":"f1c4731a-1109-4eea-8300-2a38127ae1ff","Type":"ContainerStarted","Data":"4c6d503bfa80cd3462bc3453114a0c4037cc4cd4463a1cb97eb8a1413365f43f"} Aug 01 01:12:21.099498 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:21.099327 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" event={"ID":"f1c4731a-1109-4eea-8300-2a38127ae1ff","Type":"ContainerStarted","Data":"8218eb50eb961ed2a71c7671b79880259eb6736645a5ffc370cd818aaf5901a2"} Aug 01 01:12:21.119887 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:21.119844 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" podStartSLOduration=1.119830889 podStartE2EDuration="1.119830889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:12:21.117127984 +0000 UTC m=+2629.905849825" watchObservedRunningTime="2026-08-01 01:12:21.119830889 +0000 UTC m=+2629.908552717" Aug 01 01:12:21.427847 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:21.427762 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:21.432742 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:21.432717 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:22.104217 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:22.104187 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:22.105105 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:22.105083 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:24.516966 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.516937 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:12:24.620076 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620032 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-podinfo\") pod \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " Aug 01 01:12:24.620241 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620085 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-token\") pod \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " Aug 01 01:12:24.620241 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620109 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-credential-socket\") pod \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " Aug 01 01:12:24.620241 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620125 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istiod-ca-cert\") pod \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " Aug 01 01:12:24.620241 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620162 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-workload-certs\") pod \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " Aug 01 01:12:24.620241 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620208 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjjq6\" (UniqueName: \"kubernetes.io/projected/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-kube-api-access-tjjq6\") pod \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " Aug 01 01:12:24.620241 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620228 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-workload-socket\") pod \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " Aug 01 01:12:24.620547 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620267 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-envoy\") pod \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " Aug 01 01:12:24.620547 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620304 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-data\") pod \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\" (UID: \"33f0d8cc-6d41-4ba3-8895-a40cc97a1096\") " Aug 01 01:12:24.620547 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620459 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "33f0d8cc-6d41-4ba3-8895-a40cc97a1096" (UID: "33f0d8cc-6d41-4ba3-8895-a40cc97a1096"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:12:24.620547 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620478 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "33f0d8cc-6d41-4ba3-8895-a40cc97a1096" (UID: "33f0d8cc-6d41-4ba3-8895-a40cc97a1096"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:12:24.620753 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620562 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "33f0d8cc-6d41-4ba3-8895-a40cc97a1096" (UID: "33f0d8cc-6d41-4ba3-8895-a40cc97a1096"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:12:24.620753 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620682 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "33f0d8cc-6d41-4ba3-8895-a40cc97a1096" (UID: "33f0d8cc-6d41-4ba3-8895-a40cc97a1096"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:12:24.620753 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620694 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-credential-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:24.620753 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620728 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istiod-ca-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:24.620753 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620743 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-workload-certs\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:24.620753 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.620694 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-data" (OuterVolumeSpecName: "istio-data") pod "33f0d8cc-6d41-4ba3-8895-a40cc97a1096" (UID: "33f0d8cc-6d41-4ba3-8895-a40cc97a1096"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:12:24.622400 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.622376 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "33f0d8cc-6d41-4ba3-8895-a40cc97a1096" (UID: "33f0d8cc-6d41-4ba3-8895-a40cc97a1096"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:12:24.622598 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.622566 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-kube-api-access-tjjq6" (OuterVolumeSpecName: "kube-api-access-tjjq6") pod "33f0d8cc-6d41-4ba3-8895-a40cc97a1096" (UID: "33f0d8cc-6d41-4ba3-8895-a40cc97a1096"). InnerVolumeSpecName "kube-api-access-tjjq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:12:24.622712 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.622629 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-token" (OuterVolumeSpecName: "istio-token") pod "33f0d8cc-6d41-4ba3-8895-a40cc97a1096" (UID: "33f0d8cc-6d41-4ba3-8895-a40cc97a1096"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:12:24.622712 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.622631 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "33f0d8cc-6d41-4ba3-8895-a40cc97a1096" (UID: "33f0d8cc-6d41-4ba3-8895-a40cc97a1096"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 01 01:12:24.722191 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.722152 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjjq6\" (UniqueName: \"kubernetes.io/projected/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-kube-api-access-tjjq6\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:24.722191 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.722189 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-workload-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:24.722191 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.722204 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-envoy\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:24.722518 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.722217 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-data\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:24.722518 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.722230 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-podinfo\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:24.722518 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.722243 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0d8cc-6d41-4ba3-8895-a40cc97a1096-istio-token\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:12:24.724051 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.724017 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm"] Aug 01 01:12:24.724398 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.724384 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerName="istio-proxy" Aug 01 01:12:24.724447 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.724400 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerName="istio-proxy" Aug 01 01:12:24.724447 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.724415 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerName="istio-proxy" Aug 01 01:12:24.724447 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.724421 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerName="istio-proxy" Aug 01 01:12:24.724447 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.724439 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerName="istio-proxy" Aug 01 01:12:24.724447 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.724446 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerName="istio-proxy" Aug 01 01:12:24.724592 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.724506 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a95b6023-80b6-4fef-ae57-5529cdb8b650" containerName="istio-proxy" Aug 01 01:12:24.724592 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.724515 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerName="istio-proxy" Aug 01 01:12:24.724592 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.724521 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerName="istio-proxy" Aug 01 01:12:24.728210 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.728190 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" Aug 01 01:12:24.730784 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.730764 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3b6f87-dockercfg-rjzjm\"" Aug 01 01:12:24.730896 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.730771 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3b6f87\"" Aug 01 01:12:24.736092 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.735794 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm"] Aug 01 01:12:24.773077 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.773032 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj"] Aug 01 01:12:24.776453 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.776433 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" Aug 01 01:12:24.785127 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.785105 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj"] Aug 01 01:12:24.822933 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.822896 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxns\" (UniqueName: \"kubernetes.io/projected/ec23a91f-b0c7-4211-801f-571530f27a61-kube-api-access-chxns\") pod \"payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj\" (UID: \"ec23a91f-b0c7-4211-801f-571530f27a61\") " pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" Aug 01 01:12:24.823122 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.822954 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794e461d-ac8d-49d7-b4ab-13563d90ca94-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm\" (UID: \"794e461d-ac8d-49d7-b4ab-13563d90ca94\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" Aug 01 01:12:24.823122 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.823023 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec23a91f-b0c7-4211-801f-571530f27a61-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj\" (UID: \"ec23a91f-b0c7-4211-801f-571530f27a61\") " pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" Aug 01 01:12:24.823122 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.823049 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rx6\" (UniqueName: \"kubernetes.io/projected/794e461d-ac8d-49d7-b4ab-13563d90ca94-kube-api-access-75rx6\") pod \"payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm\" (UID: \"794e461d-ac8d-49d7-b4ab-13563d90ca94\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" Aug 01 01:12:24.924305 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.924211 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794e461d-ac8d-49d7-b4ab-13563d90ca94-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm\" (UID: \"794e461d-ac8d-49d7-b4ab-13563d90ca94\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" Aug 01 01:12:24.924305 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.924291 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec23a91f-b0c7-4211-801f-571530f27a61-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj\" (UID: \"ec23a91f-b0c7-4211-801f-571530f27a61\") " pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" Aug 01 01:12:24.924535 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.924318 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rx6\" (UniqueName: \"kubernetes.io/projected/794e461d-ac8d-49d7-b4ab-13563d90ca94-kube-api-access-75rx6\") pod \"payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm\" (UID: \"794e461d-ac8d-49d7-b4ab-13563d90ca94\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" Aug 01 01:12:24.924535 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.924397 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxns\" (UniqueName: \"kubernetes.io/projected/ec23a91f-b0c7-4211-801f-571530f27a61-kube-api-access-chxns\") pod \"payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj\" (UID: \"ec23a91f-b0c7-4211-801f-571530f27a61\") " pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" Aug 01 01:12:24.925075 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.925035 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794e461d-ac8d-49d7-b4ab-13563d90ca94-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm\" (UID: \"794e461d-ac8d-49d7-b4ab-13563d90ca94\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" Aug 01 01:12:24.925181 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.925075 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec23a91f-b0c7-4211-801f-571530f27a61-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj\" (UID: \"ec23a91f-b0c7-4211-801f-571530f27a61\") " pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" Aug 01 01:12:24.931964 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.931944 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxns\" (UniqueName: \"kubernetes.io/projected/ec23a91f-b0c7-4211-801f-571530f27a61-kube-api-access-chxns\") pod \"payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj\" (UID: \"ec23a91f-b0c7-4211-801f-571530f27a61\") " pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" Aug 01 01:12:24.932042 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:24.932002 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rx6\" (UniqueName: \"kubernetes.io/projected/794e461d-ac8d-49d7-b4ab-13563d90ca94-kube-api-access-75rx6\") pod \"payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm\" (UID: \"794e461d-ac8d-49d7-b4ab-13563d90ca94\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" Aug 01 01:12:25.039895 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.039862 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" Aug 01 01:12:25.087845 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.087807 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" Aug 01 01:12:25.122750 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.122707 2563 generic.go:356] "Generic (PLEG): container finished" podID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" containerID="2126dd57e3203632f6d35ab872f966ea91efac3676434fd9a7252ea55fb54245" exitCode=0 Aug 01 01:12:25.122750 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.122744 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" event={"ID":"33f0d8cc-6d41-4ba3-8895-a40cc97a1096","Type":"ContainerDied","Data":"2126dd57e3203632f6d35ab872f966ea91efac3676434fd9a7252ea55fb54245"} Aug 01 01:12:25.122991 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.122768 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" Aug 01 01:12:25.122991 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.122816 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw" event={"ID":"33f0d8cc-6d41-4ba3-8895-a40cc97a1096","Type":"ContainerDied","Data":"f191912d8319a1e7957c48d430c4490e04300730640356ca3da2e40c87b98cc3"} Aug 01 01:12:25.122991 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.122849 2563 scope.go:117] "RemoveContainer" containerID="2126dd57e3203632f6d35ab872f966ea91efac3676434fd9a7252ea55fb54245" Aug 01 01:12:25.139442 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.139415 2563 scope.go:117] "RemoveContainer" containerID="3cade66f9648cdd04337266ff210e11bdd4b0f85c0cda14393ee5c7f2cd53af5" Aug 01 01:12:25.153012 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.152984 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw"] Aug 01 01:12:25.156113 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.153315 2563 scope.go:117] "RemoveContainer" containerID="2126dd57e3203632f6d35ab872f966ea91efac3676434fd9a7252ea55fb54245" Aug 01 01:12:25.156738 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:12:25.156648 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2126dd57e3203632f6d35ab872f966ea91efac3676434fd9a7252ea55fb54245\": container with ID starting with 2126dd57e3203632f6d35ab872f966ea91efac3676434fd9a7252ea55fb54245 not found: ID does not exist" containerID="2126dd57e3203632f6d35ab872f966ea91efac3676434fd9a7252ea55fb54245" Aug 01 01:12:25.156738 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.156675 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-061b54-openshift-default-5f95dc9f44-2rvnw"] Aug 01 01:12:25.156738 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.156688 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2126dd57e3203632f6d35ab872f966ea91efac3676434fd9a7252ea55fb54245"} err="failed to get container status \"2126dd57e3203632f6d35ab872f966ea91efac3676434fd9a7252ea55fb54245\": rpc error: code = NotFound desc = could not find container \"2126dd57e3203632f6d35ab872f966ea91efac3676434fd9a7252ea55fb54245\": container with ID starting with 2126dd57e3203632f6d35ab872f966ea91efac3676434fd9a7252ea55fb54245 not found: ID does not exist" Aug 01 01:12:25.156738 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.156718 2563 scope.go:117] "RemoveContainer" containerID="3cade66f9648cdd04337266ff210e11bdd4b0f85c0cda14393ee5c7f2cd53af5" Aug 01 01:12:25.157018 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:12:25.156995 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cade66f9648cdd04337266ff210e11bdd4b0f85c0cda14393ee5c7f2cd53af5\": container with ID starting with 3cade66f9648cdd04337266ff210e11bdd4b0f85c0cda14393ee5c7f2cd53af5 not found: ID does not exist" containerID="3cade66f9648cdd04337266ff210e11bdd4b0f85c0cda14393ee5c7f2cd53af5" Aug 01 01:12:25.157105 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.157017 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cade66f9648cdd04337266ff210e11bdd4b0f85c0cda14393ee5c7f2cd53af5"} err="failed to get container status \"3cade66f9648cdd04337266ff210e11bdd4b0f85c0cda14393ee5c7f2cd53af5\": rpc error: code = NotFound desc = could not find container \"3cade66f9648cdd04337266ff210e11bdd4b0f85c0cda14393ee5c7f2cd53af5\": container with ID starting with 3cade66f9648cdd04337266ff210e11bdd4b0f85c0cda14393ee5c7f2cd53af5 not found: ID does not exist" Aug 01 01:12:25.196307 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.195906 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm"] Aug 01 01:12:25.199010 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:12:25.198976 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794e461d_ac8d_49d7_b4ab_13563d90ca94.slice/crio-da652de17669fdf4aba730666140be67e1a23ecc627031908ad9b89dc1e81728 WatchSource:0}: Error finding container da652de17669fdf4aba730666140be67e1a23ecc627031908ad9b89dc1e81728: Status 404 returned error can't find the container with id da652de17669fdf4aba730666140be67e1a23ecc627031908ad9b89dc1e81728 Aug 01 01:12:25.244690 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.244661 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj"] Aug 01 01:12:25.259227 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:12:25.259202 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec23a91f_b0c7_4211_801f_571530f27a61.slice/crio-1ba38945d2e5b7b322a49c54002b1d16dbb4f4ecef80fa9bbfb2eea843294bc2 WatchSource:0}: Error finding container 1ba38945d2e5b7b322a49c54002b1d16dbb4f4ecef80fa9bbfb2eea843294bc2: Status 404 returned error can't find the container with id 1ba38945d2e5b7b322a49c54002b1d16dbb4f4ecef80fa9bbfb2eea843294bc2 Aug 01 01:12:25.729002 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:25.728970 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33f0d8cc-6d41-4ba3-8895-a40cc97a1096" path="/var/lib/kubelet/pods/33f0d8cc-6d41-4ba3-8895-a40cc97a1096/volumes" Aug 01 01:12:26.131710 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:26.131675 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" event={"ID":"ec23a91f-b0c7-4211-801f-571530f27a61","Type":"ContainerStarted","Data":"5d2490428b6364e9cc78d0ebbf55e4fc7fc9eedabc1be3e4f47b698106e8cb06"} Aug 01 01:12:26.131875 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:26.131729 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" event={"ID":"ec23a91f-b0c7-4211-801f-571530f27a61","Type":"ContainerStarted","Data":"1ba38945d2e5b7b322a49c54002b1d16dbb4f4ecef80fa9bbfb2eea843294bc2"} Aug 01 01:12:26.131875 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:26.131788 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" Aug 01 01:12:26.133307 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:26.133281 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" event={"ID":"794e461d-ac8d-49d7-b4ab-13563d90ca94","Type":"ContainerStarted","Data":"63afc37c65ad8a2feaf1fa8438a4f3985655ad31b9425f9d1ba1493e6bd969b7"} Aug 01 01:12:26.133414 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:26.133315 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" event={"ID":"794e461d-ac8d-49d7-b4ab-13563d90ca94","Type":"ContainerStarted","Data":"da652de17669fdf4aba730666140be67e1a23ecc627031908ad9b89dc1e81728"} Aug 01 01:12:26.133414 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:26.133390 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" Aug 01 01:12:26.151825 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:26.151779 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" podStartSLOduration=2.15176163 podStartE2EDuration="2.15176163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:12:26.149193242 +0000 UTC m=+2634.937915072" watchObservedRunningTime="2026-08-01 01:12:26.15176163 +0000 UTC m=+2634.940483457" Aug 01 01:12:26.165583 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:26.165519 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" podStartSLOduration=2.165501071 podStartE2EDuration="2.165501071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:12:26.163249114 +0000 UTC m=+2634.951970942" watchObservedRunningTime="2026-08-01 01:12:26.165501071 +0000 UTC m=+2634.954222897" Aug 01 01:12:37.140241 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:37.140210 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" Aug 01 01:12:37.140621 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:37.140280 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" Aug 01 01:12:38.145207 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:38.145174 2563 patch_prober.go:28] interesting pod/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 01 01:12:38.145600 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:38.145237 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 01 01:12:39.200342 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:39.200312 2563 generic.go:356] "Generic (PLEG): container finished" podID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerID="4c6d503bfa80cd3462bc3453114a0c4037cc4cd4463a1cb97eb8a1413365f43f" exitCode=0 Aug 01 01:12:39.200792 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:39.200374 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" event={"ID":"f1c4731a-1109-4eea-8300-2a38127ae1ff","Type":"ContainerDied","Data":"4c6d503bfa80cd3462bc3453114a0c4037cc4cd4463a1cb97eb8a1413365f43f"} Aug 01 01:12:39.200876 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:39.200798 2563 scope.go:117] "RemoveContainer" containerID="4c6d503bfa80cd3462bc3453114a0c4037cc4cd4463a1cb97eb8a1413365f43f" Aug 01 01:12:39.204631 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:39.202912 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:12:39.204631 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:39.203391 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:12:39.204631 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:39.204007 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:12:40.207014 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:40.206982 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" event={"ID":"f1c4731a-1109-4eea-8300-2a38127ae1ff","Type":"ContainerStarted","Data":"407c8b6c09336cdad26aea3e4c87202bdb9c7ce1aa75d22eb1fa899a1f8f7cfc"} Aug 01 01:12:40.428364 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:40.428259 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:40.428718 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:40.428675 2563 patch_prober.go:28] interesting pod/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Aug 01 01:12:40.428823 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:40.428737 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Aug 01 01:12:41.430085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:41.428765 2563 patch_prober.go:28] interesting pod/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Aug 01 01:12:41.430085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:41.428837 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Aug 01 01:12:42.428024 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:42.427987 2563 patch_prober.go:28] interesting pod/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Aug 01 01:12:42.428358 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:42.428329 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Aug 01 01:12:43.428074 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:43.428018 2563 patch_prober.go:28] interesting pod/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Aug 01 01:12:43.428627 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:43.428108 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Aug 01 01:12:44.428340 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:44.428300 2563 patch_prober.go:28] interesting pod/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Aug 01 01:12:44.429047 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:44.429015 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Aug 01 01:12:45.431463 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:45.431427 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:45.431841 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:45.431818 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:45.432496 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:45.432478 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:12:48.671280 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.671244 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m"] Aug 01 01:12:48.675044 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.675023 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.678194 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.678175 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-113a68-openshift-default-dockercfg-qpldq\"" Aug 01 01:12:48.688872 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.688851 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m"] Aug 01 01:12:48.758704 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.758671 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-envoy\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.758704 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.758708 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-podinfo\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.758896 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.758734 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istiod-ca-cert\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.758896 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.758833 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-workload-socket\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.758896 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.758880 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-token\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.759002 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.758964 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-data\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.759002 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.758990 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-credential-socket\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.759096 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.759016 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-workload-certs\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.759138 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.759094 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c5gd\" (UniqueName: \"kubernetes.io/projected/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-kube-api-access-5c5gd\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.860356 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.860321 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c5gd\" (UniqueName: \"kubernetes.io/projected/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-kube-api-access-5c5gd\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.860547 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.860387 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-envoy\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.860547 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.860414 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-podinfo\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.860547 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.860458 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istiod-ca-cert\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.860547 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.860531 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-workload-socket\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.860775 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.860568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-token\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.860775 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.860679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-data\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.860775 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.860705 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-credential-socket\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.860775 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.860744 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-workload-certs\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.861431 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.861120 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-workload-socket\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.861431 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.861137 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-data\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.861431 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.861313 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-credential-socket\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.861431 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.861334 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-workload-certs\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.861431 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.861373 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istiod-ca-cert\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.862885 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.862852 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-envoy\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.863526 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.863501 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-podinfo\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.869146 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.869124 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-token\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.869377 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.869359 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c5gd\" (UniqueName: \"kubernetes.io/projected/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-kube-api-access-5c5gd\") pod \"e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:48.988941 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:48.988859 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:49.125696 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:49.125666 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m"] Aug 01 01:12:49.129074 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:12:49.129036 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db2f9d0_627e_4dd4_8b5f_f8455c8a0426.slice/crio-9392cd73c4fa63a4be9dfc62d9e7bab8a5623d81a2bec5223f9b68ec83e69c46 WatchSource:0}: Error finding container 9392cd73c4fa63a4be9dfc62d9e7bab8a5623d81a2bec5223f9b68ec83e69c46: Status 404 returned error can't find the container with id 9392cd73c4fa63a4be9dfc62d9e7bab8a5623d81a2bec5223f9b68ec83e69c46 Aug 01 01:12:49.131700 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:49.131656 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:12:49.131813 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:49.131738 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:12:49.131813 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:49.131782 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:12:49.264893 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:49.264806 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" event={"ID":"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426","Type":"ContainerStarted","Data":"c695789161b0ffd7fc9be8f6bb8152b21d53343a992263bb5bd73a65a7212625"} Aug 01 01:12:49.264893 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:49.264859 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" event={"ID":"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426","Type":"ContainerStarted","Data":"9392cd73c4fa63a4be9dfc62d9e7bab8a5623d81a2bec5223f9b68ec83e69c46"} Aug 01 01:12:49.285868 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:49.285806 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" podStartSLOduration=1.285790525 podStartE2EDuration="1.285790525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:12:49.283192087 +0000 UTC m=+2658.071913920" watchObservedRunningTime="2026-08-01 01:12:49.285790525 +0000 UTC m=+2658.074512353" Aug 01 01:12:49.989957 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:49.989923 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:50.994745 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:50.994712 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:51.275781 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:51.275701 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:51.276731 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:51.276710 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:12:52.999085 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:52.999027 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q"] Aug 01 01:12:53.002697 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:53.002680 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" Aug 01 01:12:53.005198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:53.005175 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-113a68\"" Aug 01 01:12:53.005298 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:53.005255 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-113a68-dockercfg-vnxnr\"" Aug 01 01:12:53.009951 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:53.009925 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q"] Aug 01 01:12:53.098880 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:53.098839 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78r6w\" (UniqueName: \"kubernetes.io/projected/f57922bb-a3bb-4783-a4a4-a86a54e419b4-kube-api-access-78r6w\") pod \"payload-processing-e2e-api-b-113a68-7547bb9498-j652q\" (UID: \"f57922bb-a3bb-4783-a4a4-a86a54e419b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" Aug 01 01:12:53.099075 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:53.098918 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57922bb-a3bb-4783-a4a4-a86a54e419b4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-113a68-7547bb9498-j652q\" (UID: \"f57922bb-a3bb-4783-a4a4-a86a54e419b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" Aug 01 01:12:53.200121 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:53.200083 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57922bb-a3bb-4783-a4a4-a86a54e419b4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-113a68-7547bb9498-j652q\" (UID: \"f57922bb-a3bb-4783-a4a4-a86a54e419b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" Aug 01 01:12:53.200311 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:53.200226 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78r6w\" (UniqueName: \"kubernetes.io/projected/f57922bb-a3bb-4783-a4a4-a86a54e419b4-kube-api-access-78r6w\") pod \"payload-processing-e2e-api-b-113a68-7547bb9498-j652q\" (UID: \"f57922bb-a3bb-4783-a4a4-a86a54e419b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" Aug 01 01:12:53.200805 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:53.200784 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57922bb-a3bb-4783-a4a4-a86a54e419b4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-113a68-7547bb9498-j652q\" (UID: \"f57922bb-a3bb-4783-a4a4-a86a54e419b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" Aug 01 01:12:53.217256 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:53.217227 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78r6w\" (UniqueName: \"kubernetes.io/projected/f57922bb-a3bb-4783-a4a4-a86a54e419b4-kube-api-access-78r6w\") pod \"payload-processing-e2e-api-b-113a68-7547bb9498-j652q\" (UID: \"f57922bb-a3bb-4783-a4a4-a86a54e419b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" Aug 01 01:12:53.316169 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:53.316144 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" Aug 01 01:12:53.460437 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:53.460412 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q"] Aug 01 01:12:53.462791 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:12:53.462758 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57922bb_a3bb_4783_a4a4_a86a54e419b4.slice/crio-027c9aa72c8534be806df964ed3493388a12b1060c44edfa0b48bb64d5e7cb73 WatchSource:0}: Error finding container 027c9aa72c8534be806df964ed3493388a12b1060c44edfa0b48bb64d5e7cb73: Status 404 returned error can't find the container with id 027c9aa72c8534be806df964ed3493388a12b1060c44edfa0b48bb64d5e7cb73 Aug 01 01:12:54.292512 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:54.292479 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" event={"ID":"f57922bb-a3bb-4783-a4a4-a86a54e419b4","Type":"ContainerStarted","Data":"0901f4d8a5d6ab90e39ac0bae05ab9ee33d42150ff5f562e750021bb4c305183"} Aug 01 01:12:54.292999 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:54.292528 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" event={"ID":"f57922bb-a3bb-4783-a4a4-a86a54e419b4","Type":"ContainerStarted","Data":"027c9aa72c8534be806df964ed3493388a12b1060c44edfa0b48bb64d5e7cb73"} Aug 01 01:12:54.292999 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:54.292608 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" Aug 01 01:12:54.308559 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:12:54.308519 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" podStartSLOduration=2.308503989 podStartE2EDuration="2.308503989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:12:54.307091123 +0000 UTC m=+2663.095812952" watchObservedRunningTime="2026-08-01 01:12:54.308503989 +0000 UTC m=+2663.097225815" Aug 01 01:13:05.299807 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:05.299774 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" Aug 01 01:13:06.276681 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:06.276648 2563 patch_prober.go:28] interesting pod/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Aug 01 01:13:06.276892 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:06.276708 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Aug 01 01:13:06.361414 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:06.361388 2563 generic.go:356] "Generic (PLEG): container finished" podID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerID="c695789161b0ffd7fc9be8f6bb8152b21d53343a992263bb5bd73a65a7212625" exitCode=0 Aug 01 01:13:06.361751 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:06.361466 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" event={"ID":"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426","Type":"ContainerDied","Data":"c695789161b0ffd7fc9be8f6bb8152b21d53343a992263bb5bd73a65a7212625"} Aug 01 01:13:06.361839 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:06.361825 2563 scope.go:117] "RemoveContainer" containerID="c695789161b0ffd7fc9be8f6bb8152b21d53343a992263bb5bd73a65a7212625" Aug 01 01:13:06.362834 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:06.362803 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:13:06.362914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:06.362897 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:13:06.362952 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:06.362939 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:13:07.368509 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:07.368471 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" event={"ID":"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426","Type":"ContainerStarted","Data":"ac8f9e531c6ce357b78f40d6c062015358456166022330bd7725c026a9ffcf24"} Aug 01 01:13:07.989759 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:07.989714 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:13:07.989951 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:07.989928 2563 patch_prober.go:28] interesting pod/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Aug 01 01:13:07.990006 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:07.989987 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Aug 01 01:13:08.989394 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:08.989363 2563 patch_prober.go:28] interesting pod/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Aug 01 01:13:08.989769 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:08.989425 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Aug 01 01:13:09.989739 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:09.989699 2563 patch_prober.go:28] interesting pod/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Aug 01 01:13:09.990160 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:09.989760 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Aug 01 01:13:10.989770 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:10.989731 2563 patch_prober.go:28] interesting pod/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Aug 01 01:13:10.990290 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:10.989804 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Aug 01 01:13:11.989534 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:11.989498 2563 patch_prober.go:28] interesting pod/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Aug 01 01:13:11.989714 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:11.989552 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Aug 01 01:13:12.993293 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:12.993260 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:13:12.993676 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:12.993624 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:13:12.994309 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:12.994292 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:13:31.873711 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:31.873637 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 01:13:31.883046 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:31.883022 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 01:13:31.883522 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:31.883505 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 01:13:31.888926 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:31.888908 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 01:13:34.437201 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.437165 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj"] Aug 01 01:13:34.437638 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.437413 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" podUID="ec23a91f-b0c7-4211-801f-571530f27a61" containerName="payload-processing" containerID="cri-o://5d2490428b6364e9cc78d0ebbf55e4fc7fc9eedabc1be3e4f47b698106e8cb06" gracePeriod=30 Aug 01 01:13:34.443748 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.443720 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm"] Aug 01 01:13:34.443988 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.443948 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" podUID="794e461d-ac8d-49d7-b4ab-13563d90ca94" containerName="payload-pre-processing" containerID="cri-o://63afc37c65ad8a2feaf1fa8438a4f3985655ad31b9425f9d1ba1493e6bd969b7" gracePeriod=30 Aug 01 01:13:34.699412 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.699356 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" Aug 01 01:13:34.730414 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.730393 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" Aug 01 01:13:34.796591 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.796570 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75rx6\" (UniqueName: \"kubernetes.io/projected/794e461d-ac8d-49d7-b4ab-13563d90ca94-kube-api-access-75rx6\") pod \"794e461d-ac8d-49d7-b4ab-13563d90ca94\" (UID: \"794e461d-ac8d-49d7-b4ab-13563d90ca94\") " Aug 01 01:13:34.796724 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.796615 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec23a91f-b0c7-4211-801f-571530f27a61-plugins-config-volume\") pod \"ec23a91f-b0c7-4211-801f-571530f27a61\" (UID: \"ec23a91f-b0c7-4211-801f-571530f27a61\") " Aug 01 01:13:34.796724 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.796695 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chxns\" (UniqueName: \"kubernetes.io/projected/ec23a91f-b0c7-4211-801f-571530f27a61-kube-api-access-chxns\") pod \"ec23a91f-b0c7-4211-801f-571530f27a61\" (UID: \"ec23a91f-b0c7-4211-801f-571530f27a61\") " Aug 01 01:13:34.796803 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.796761 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794e461d-ac8d-49d7-b4ab-13563d90ca94-plugins-config-volume\") pod \"794e461d-ac8d-49d7-b4ab-13563d90ca94\" (UID: \"794e461d-ac8d-49d7-b4ab-13563d90ca94\") " Aug 01 01:13:34.797013 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.796977 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec23a91f-b0c7-4211-801f-571530f27a61-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec23a91f-b0c7-4211-801f-571530f27a61" (UID: "ec23a91f-b0c7-4211-801f-571530f27a61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:13:34.797167 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.797132 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794e461d-ac8d-49d7-b4ab-13563d90ca94-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "794e461d-ac8d-49d7-b4ab-13563d90ca94" (UID: "794e461d-ac8d-49d7-b4ab-13563d90ca94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:13:34.797402 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.797380 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec23a91f-b0c7-4211-801f-571530f27a61-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:34.797529 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.797406 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794e461d-ac8d-49d7-b4ab-13563d90ca94-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:34.798721 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.798700 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794e461d-ac8d-49d7-b4ab-13563d90ca94-kube-api-access-75rx6" (OuterVolumeSpecName: "kube-api-access-75rx6") pod "794e461d-ac8d-49d7-b4ab-13563d90ca94" (UID: "794e461d-ac8d-49d7-b4ab-13563d90ca94"). InnerVolumeSpecName "kube-api-access-75rx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:13:34.798802 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.798737 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec23a91f-b0c7-4211-801f-571530f27a61-kube-api-access-chxns" (OuterVolumeSpecName: "kube-api-access-chxns") pod "ec23a91f-b0c7-4211-801f-571530f27a61" (UID: "ec23a91f-b0c7-4211-801f-571530f27a61"). InnerVolumeSpecName "kube-api-access-chxns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:13:34.898674 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.898647 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rx6\" (UniqueName: \"kubernetes.io/projected/794e461d-ac8d-49d7-b4ab-13563d90ca94-kube-api-access-75rx6\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:34.898674 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:34.898670 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chxns\" (UniqueName: \"kubernetes.io/projected/ec23a91f-b0c7-4211-801f-571530f27a61-kube-api-access-chxns\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:35.514266 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.514235 2563 generic.go:356] "Generic (PLEG): container finished" podID="794e461d-ac8d-49d7-b4ab-13563d90ca94" containerID="63afc37c65ad8a2feaf1fa8438a4f3985655ad31b9425f9d1ba1493e6bd969b7" exitCode=0 Aug 01 01:13:35.514698 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.514289 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" Aug 01 01:13:35.514698 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.514323 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" event={"ID":"794e461d-ac8d-49d7-b4ab-13563d90ca94","Type":"ContainerDied","Data":"63afc37c65ad8a2feaf1fa8438a4f3985655ad31b9425f9d1ba1493e6bd969b7"} Aug 01 01:13:35.514698 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.514397 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm" event={"ID":"794e461d-ac8d-49d7-b4ab-13563d90ca94","Type":"ContainerDied","Data":"da652de17669fdf4aba730666140be67e1a23ecc627031908ad9b89dc1e81728"} Aug 01 01:13:35.514698 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.514428 2563 scope.go:117] "RemoveContainer" containerID="63afc37c65ad8a2feaf1fa8438a4f3985655ad31b9425f9d1ba1493e6bd969b7" Aug 01 01:13:35.515965 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.515933 2563 generic.go:356] "Generic (PLEG): container finished" podID="ec23a91f-b0c7-4211-801f-571530f27a61" containerID="5d2490428b6364e9cc78d0ebbf55e4fc7fc9eedabc1be3e4f47b698106e8cb06" exitCode=0 Aug 01 01:13:35.516155 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.515964 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" event={"ID":"ec23a91f-b0c7-4211-801f-571530f27a61","Type":"ContainerDied","Data":"5d2490428b6364e9cc78d0ebbf55e4fc7fc9eedabc1be3e4f47b698106e8cb06"} Aug 01 01:13:35.516155 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.515989 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" Aug 01 01:13:35.516155 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.516015 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj" event={"ID":"ec23a91f-b0c7-4211-801f-571530f27a61","Type":"ContainerDied","Data":"1ba38945d2e5b7b322a49c54002b1d16dbb4f4ecef80fa9bbfb2eea843294bc2"} Aug 01 01:13:35.524633 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.524615 2563 scope.go:117] "RemoveContainer" containerID="63afc37c65ad8a2feaf1fa8438a4f3985655ad31b9425f9d1ba1493e6bd969b7" Aug 01 01:13:35.524929 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:13:35.524902 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63afc37c65ad8a2feaf1fa8438a4f3985655ad31b9425f9d1ba1493e6bd969b7\": container with ID starting with 63afc37c65ad8a2feaf1fa8438a4f3985655ad31b9425f9d1ba1493e6bd969b7 not found: ID does not exist" containerID="63afc37c65ad8a2feaf1fa8438a4f3985655ad31b9425f9d1ba1493e6bd969b7" Aug 01 01:13:35.525043 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.524962 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63afc37c65ad8a2feaf1fa8438a4f3985655ad31b9425f9d1ba1493e6bd969b7"} err="failed to get container status \"63afc37c65ad8a2feaf1fa8438a4f3985655ad31b9425f9d1ba1493e6bd969b7\": rpc error: code = NotFound desc = could not find container \"63afc37c65ad8a2feaf1fa8438a4f3985655ad31b9425f9d1ba1493e6bd969b7\": container with ID starting with 63afc37c65ad8a2feaf1fa8438a4f3985655ad31b9425f9d1ba1493e6bd969b7 not found: ID does not exist" Aug 01 01:13:35.525043 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.524991 2563 scope.go:117] "RemoveContainer" containerID="5d2490428b6364e9cc78d0ebbf55e4fc7fc9eedabc1be3e4f47b698106e8cb06" Aug 01 01:13:35.533302 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.533286 2563 scope.go:117] "RemoveContainer" containerID="5d2490428b6364e9cc78d0ebbf55e4fc7fc9eedabc1be3e4f47b698106e8cb06" Aug 01 01:13:35.533551 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:13:35.533531 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2490428b6364e9cc78d0ebbf55e4fc7fc9eedabc1be3e4f47b698106e8cb06\": container with ID starting with 5d2490428b6364e9cc78d0ebbf55e4fc7fc9eedabc1be3e4f47b698106e8cb06 not found: ID does not exist" containerID="5d2490428b6364e9cc78d0ebbf55e4fc7fc9eedabc1be3e4f47b698106e8cb06" Aug 01 01:13:35.533609 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.533560 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2490428b6364e9cc78d0ebbf55e4fc7fc9eedabc1be3e4f47b698106e8cb06"} err="failed to get container status \"5d2490428b6364e9cc78d0ebbf55e4fc7fc9eedabc1be3e4f47b698106e8cb06\": rpc error: code = NotFound desc = could not find container \"5d2490428b6364e9cc78d0ebbf55e4fc7fc9eedabc1be3e4f47b698106e8cb06\": container with ID starting with 5d2490428b6364e9cc78d0ebbf55e4fc7fc9eedabc1be3e4f47b698106e8cb06 not found: ID does not exist" Aug 01 01:13:35.541669 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.541644 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm"] Aug 01 01:13:35.545249 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.545223 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3b6f87-7897d66c4d-g7rmm"] Aug 01 01:13:35.554966 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.554938 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj"] Aug 01 01:13:35.558627 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.558605 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3b6f87-85599768c6-kd5pj"] Aug 01 01:13:35.728726 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.728648 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="794e461d-ac8d-49d7-b4ab-13563d90ca94" path="/var/lib/kubelet/pods/794e461d-ac8d-49d7-b4ab-13563d90ca94/volumes" Aug 01 01:13:35.729017 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:35.729004 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec23a91f-b0c7-4211-801f-571530f27a61" path="/var/lib/kubelet/pods/ec23a91f-b0c7-4211-801f-571530f27a61/volumes" Aug 01 01:13:40.920710 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:40.920670 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf"] Aug 01 01:13:40.921268 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:40.920895 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" containerID="cri-o://407c8b6c09336cdad26aea3e4c87202bdb9c7ce1aa75d22eb1fa899a1f8f7cfc" gracePeriod=30 Aug 01 01:13:45.434443 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:45.434402 2563 patch_prober.go:28] interesting pod/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 01 01:13:45.434808 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:45.434463 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 01 01:13:46.172580 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.172555 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:13:46.308920 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.308891 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-credential-socket\") pod \"f1c4731a-1109-4eea-8300-2a38127ae1ff\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " Aug 01 01:13:46.309104 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.308935 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-data\") pod \"f1c4731a-1109-4eea-8300-2a38127ae1ff\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " Aug 01 01:13:46.309104 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.308975 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c4731a-1109-4eea-8300-2a38127ae1ff-istiod-ca-cert\") pod \"f1c4731a-1109-4eea-8300-2a38127ae1ff\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " Aug 01 01:13:46.309104 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.308992 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-workload-certs\") pod \"f1c4731a-1109-4eea-8300-2a38127ae1ff\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " Aug 01 01:13:46.309104 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309014 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-envoy\") pod \"f1c4731a-1109-4eea-8300-2a38127ae1ff\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " Aug 01 01:13:46.309104 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309085 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-podinfo\") pod \"f1c4731a-1109-4eea-8300-2a38127ae1ff\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " Aug 01 01:13:46.309342 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309119 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-workload-socket\") pod \"f1c4731a-1109-4eea-8300-2a38127ae1ff\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " Aug 01 01:13:46.309342 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309162 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-token\") pod \"f1c4731a-1109-4eea-8300-2a38127ae1ff\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " Aug 01 01:13:46.309342 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309202 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntzdc\" (UniqueName: \"kubernetes.io/projected/f1c4731a-1109-4eea-8300-2a38127ae1ff-kube-api-access-ntzdc\") pod \"f1c4731a-1109-4eea-8300-2a38127ae1ff\" (UID: \"f1c4731a-1109-4eea-8300-2a38127ae1ff\") " Aug 01 01:13:46.309342 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309261 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f1c4731a-1109-4eea-8300-2a38127ae1ff" (UID: "f1c4731a-1109-4eea-8300-2a38127ae1ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:13:46.309535 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309418 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f1c4731a-1109-4eea-8300-2a38127ae1ff" (UID: "f1c4731a-1109-4eea-8300-2a38127ae1ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:13:46.309535 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309439 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-data" (OuterVolumeSpecName: "istio-data") pod "f1c4731a-1109-4eea-8300-2a38127ae1ff" (UID: "f1c4731a-1109-4eea-8300-2a38127ae1ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:13:46.309535 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309476 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c4731a-1109-4eea-8300-2a38127ae1ff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f1c4731a-1109-4eea-8300-2a38127ae1ff" (UID: "f1c4731a-1109-4eea-8300-2a38127ae1ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:13:46.309659 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309557 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-credential-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:46.309659 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309572 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-data\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:46.309659 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309586 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c4731a-1109-4eea-8300-2a38127ae1ff-istiod-ca-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:46.309659 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309598 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-workload-certs\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:46.309868 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.309824 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f1c4731a-1109-4eea-8300-2a38127ae1ff" (UID: "f1c4731a-1109-4eea-8300-2a38127ae1ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:13:46.311438 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.311417 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f1c4731a-1109-4eea-8300-2a38127ae1ff" (UID: "f1c4731a-1109-4eea-8300-2a38127ae1ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 01 01:13:46.311438 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.311420 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f1c4731a-1109-4eea-8300-2a38127ae1ff" (UID: "f1c4731a-1109-4eea-8300-2a38127ae1ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:13:46.311569 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.311437 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "f1c4731a-1109-4eea-8300-2a38127ae1ff" (UID: "f1c4731a-1109-4eea-8300-2a38127ae1ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:13:46.311569 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.311465 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c4731a-1109-4eea-8300-2a38127ae1ff-kube-api-access-ntzdc" (OuterVolumeSpecName: "kube-api-access-ntzdc") pod "f1c4731a-1109-4eea-8300-2a38127ae1ff" (UID: "f1c4731a-1109-4eea-8300-2a38127ae1ff"). InnerVolumeSpecName "kube-api-access-ntzdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:13:46.410785 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.410741 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-envoy\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:46.410785 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.410778 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-podinfo\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:46.410785 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.410786 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c4731a-1109-4eea-8300-2a38127ae1ff-workload-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:46.410785 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.410797 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c4731a-1109-4eea-8300-2a38127ae1ff-istio-token\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:46.411174 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.410808 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntzdc\" (UniqueName: \"kubernetes.io/projected/f1c4731a-1109-4eea-8300-2a38127ae1ff-kube-api-access-ntzdc\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:46.572296 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.572207 2563 generic.go:356] "Generic (PLEG): container finished" podID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerID="407c8b6c09336cdad26aea3e4c87202bdb9c7ce1aa75d22eb1fa899a1f8f7cfc" exitCode=0 Aug 01 01:13:46.572296 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.572258 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" Aug 01 01:13:46.572721 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.572291 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" event={"ID":"f1c4731a-1109-4eea-8300-2a38127ae1ff","Type":"ContainerDied","Data":"407c8b6c09336cdad26aea3e4c87202bdb9c7ce1aa75d22eb1fa899a1f8f7cfc"} Aug 01 01:13:46.572721 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.572352 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf" event={"ID":"f1c4731a-1109-4eea-8300-2a38127ae1ff","Type":"ContainerDied","Data":"8218eb50eb961ed2a71c7671b79880259eb6736645a5ffc370cd818aaf5901a2"} Aug 01 01:13:46.572721 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.572382 2563 scope.go:117] "RemoveContainer" containerID="407c8b6c09336cdad26aea3e4c87202bdb9c7ce1aa75d22eb1fa899a1f8f7cfc" Aug 01 01:13:46.581556 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.581540 2563 scope.go:117] "RemoveContainer" containerID="4c6d503bfa80cd3462bc3453114a0c4037cc4cd4463a1cb97eb8a1413365f43f" Aug 01 01:13:46.589171 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.589148 2563 scope.go:117] "RemoveContainer" containerID="407c8b6c09336cdad26aea3e4c87202bdb9c7ce1aa75d22eb1fa899a1f8f7cfc" Aug 01 01:13:46.589431 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:13:46.589411 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407c8b6c09336cdad26aea3e4c87202bdb9c7ce1aa75d22eb1fa899a1f8f7cfc\": container with ID starting with 407c8b6c09336cdad26aea3e4c87202bdb9c7ce1aa75d22eb1fa899a1f8f7cfc not found: ID does not exist" containerID="407c8b6c09336cdad26aea3e4c87202bdb9c7ce1aa75d22eb1fa899a1f8f7cfc" Aug 01 01:13:46.589477 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.589441 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407c8b6c09336cdad26aea3e4c87202bdb9c7ce1aa75d22eb1fa899a1f8f7cfc"} err="failed to get container status \"407c8b6c09336cdad26aea3e4c87202bdb9c7ce1aa75d22eb1fa899a1f8f7cfc\": rpc error: code = NotFound desc = could not find container \"407c8b6c09336cdad26aea3e4c87202bdb9c7ce1aa75d22eb1fa899a1f8f7cfc\": container with ID starting with 407c8b6c09336cdad26aea3e4c87202bdb9c7ce1aa75d22eb1fa899a1f8f7cfc not found: ID does not exist" Aug 01 01:13:46.589477 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.589461 2563 scope.go:117] "RemoveContainer" containerID="4c6d503bfa80cd3462bc3453114a0c4037cc4cd4463a1cb97eb8a1413365f43f" Aug 01 01:13:46.589688 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:13:46.589670 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6d503bfa80cd3462bc3453114a0c4037cc4cd4463a1cb97eb8a1413365f43f\": container with ID starting with 4c6d503bfa80cd3462bc3453114a0c4037cc4cd4463a1cb97eb8a1413365f43f not found: ID does not exist" containerID="4c6d503bfa80cd3462bc3453114a0c4037cc4cd4463a1cb97eb8a1413365f43f" Aug 01 01:13:46.589745 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.589692 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6d503bfa80cd3462bc3453114a0c4037cc4cd4463a1cb97eb8a1413365f43f"} err="failed to get container status \"4c6d503bfa80cd3462bc3453114a0c4037cc4cd4463a1cb97eb8a1413365f43f\": rpc error: code = NotFound desc = could not find container \"4c6d503bfa80cd3462bc3453114a0c4037cc4cd4463a1cb97eb8a1413365f43f\": container with ID starting with 4c6d503bfa80cd3462bc3453114a0c4037cc4cd4463a1cb97eb8a1413365f43f not found: ID does not exist" Aug 01 01:13:46.593969 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.593946 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf"] Aug 01 01:13:46.598359 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:46.598333 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3b6f87-openshift-default-6588b58db4-z66sf"] Aug 01 01:13:47.728583 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:47.728548 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" path="/var/lib/kubelet/pods/f1c4731a-1109-4eea-8300-2a38127ae1ff/volumes" Aug 01 01:13:57.273042 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.273005 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q"] Aug 01 01:13:57.273972 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.273918 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" podUID="f57922bb-a3bb-4783-a4a4-a86a54e419b4" containerName="payload-processing" containerID="cri-o://0901f4d8a5d6ab90e39ac0bae05ab9ee33d42150ff5f562e750021bb4c305183" gracePeriod=30 Aug 01 01:13:57.549837 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.549808 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" Aug 01 01:13:57.604401 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.604369 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57922bb-a3bb-4783-a4a4-a86a54e419b4-plugins-config-volume\") pod \"f57922bb-a3bb-4783-a4a4-a86a54e419b4\" (UID: \"f57922bb-a3bb-4783-a4a4-a86a54e419b4\") " Aug 01 01:13:57.604561 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.604411 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78r6w\" (UniqueName: \"kubernetes.io/projected/f57922bb-a3bb-4783-a4a4-a86a54e419b4-kube-api-access-78r6w\") pod \"f57922bb-a3bb-4783-a4a4-a86a54e419b4\" (UID: \"f57922bb-a3bb-4783-a4a4-a86a54e419b4\") " Aug 01 01:13:57.604793 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.604768 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57922bb-a3bb-4783-a4a4-a86a54e419b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f57922bb-a3bb-4783-a4a4-a86a54e419b4" (UID: "f57922bb-a3bb-4783-a4a4-a86a54e419b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:13:57.606511 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.606483 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57922bb-a3bb-4783-a4a4-a86a54e419b4-kube-api-access-78r6w" (OuterVolumeSpecName: "kube-api-access-78r6w") pod "f57922bb-a3bb-4783-a4a4-a86a54e419b4" (UID: "f57922bb-a3bb-4783-a4a4-a86a54e419b4"). InnerVolumeSpecName "kube-api-access-78r6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:13:57.627625 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.627593 2563 generic.go:356] "Generic (PLEG): container finished" podID="f57922bb-a3bb-4783-a4a4-a86a54e419b4" containerID="0901f4d8a5d6ab90e39ac0bae05ab9ee33d42150ff5f562e750021bb4c305183" exitCode=0 Aug 01 01:13:57.627812 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.627640 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" Aug 01 01:13:57.627812 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.627682 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" event={"ID":"f57922bb-a3bb-4783-a4a4-a86a54e419b4","Type":"ContainerDied","Data":"0901f4d8a5d6ab90e39ac0bae05ab9ee33d42150ff5f562e750021bb4c305183"} Aug 01 01:13:57.627812 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.627750 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q" event={"ID":"f57922bb-a3bb-4783-a4a4-a86a54e419b4","Type":"ContainerDied","Data":"027c9aa72c8534be806df964ed3493388a12b1060c44edfa0b48bb64d5e7cb73"} Aug 01 01:13:57.627812 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.627779 2563 scope.go:117] "RemoveContainer" containerID="0901f4d8a5d6ab90e39ac0bae05ab9ee33d42150ff5f562e750021bb4c305183" Aug 01 01:13:57.640307 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.640287 2563 scope.go:117] "RemoveContainer" containerID="0901f4d8a5d6ab90e39ac0bae05ab9ee33d42150ff5f562e750021bb4c305183" Aug 01 01:13:57.640652 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:13:57.640625 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0901f4d8a5d6ab90e39ac0bae05ab9ee33d42150ff5f562e750021bb4c305183\": container with ID starting with 0901f4d8a5d6ab90e39ac0bae05ab9ee33d42150ff5f562e750021bb4c305183 not found: ID does not exist" containerID="0901f4d8a5d6ab90e39ac0bae05ab9ee33d42150ff5f562e750021bb4c305183" Aug 01 01:13:57.640739 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.640661 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0901f4d8a5d6ab90e39ac0bae05ab9ee33d42150ff5f562e750021bb4c305183"} err="failed to get container status \"0901f4d8a5d6ab90e39ac0bae05ab9ee33d42150ff5f562e750021bb4c305183\": rpc error: code = NotFound desc = could not find container \"0901f4d8a5d6ab90e39ac0bae05ab9ee33d42150ff5f562e750021bb4c305183\": container with ID starting with 0901f4d8a5d6ab90e39ac0bae05ab9ee33d42150ff5f562e750021bb4c305183 not found: ID does not exist" Aug 01 01:13:57.659843 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.659817 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q"] Aug 01 01:13:57.666156 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.666131 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-113a68-7547bb9498-j652q"] Aug 01 01:13:57.705618 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.705588 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57922bb-a3bb-4783-a4a4-a86a54e419b4-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:57.705618 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.705613 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78r6w\" (UniqueName: \"kubernetes.io/projected/f57922bb-a3bb-4783-a4a4-a86a54e419b4-kube-api-access-78r6w\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:13:57.728946 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:13:57.728914 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f57922bb-a3bb-4783-a4a4-a86a54e419b4" path="/var/lib/kubelet/pods/f57922bb-a3bb-4783-a4a4-a86a54e419b4/volumes" Aug 01 01:14:04.039381 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.039348 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m"] Aug 01 01:14:04.039981 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.039574 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerName="istio-proxy" containerID="cri-o://ac8f9e531c6ce357b78f40d6c062015358456166022330bd7725c026a9ffcf24" gracePeriod=30 Aug 01 01:14:04.884710 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.884679 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8"] Aug 01 01:14:04.885051 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885035 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" Aug 01 01:14:04.885051 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885051 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" Aug 01 01:14:04.885198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885093 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" Aug 01 01:14:04.885198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885100 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" Aug 01 01:14:04.885198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885113 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="794e461d-ac8d-49d7-b4ab-13563d90ca94" containerName="payload-pre-processing" Aug 01 01:14:04.885198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885119 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="794e461d-ac8d-49d7-b4ab-13563d90ca94" containerName="payload-pre-processing" Aug 01 01:14:04.885198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885131 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f57922bb-a3bb-4783-a4a4-a86a54e419b4" containerName="payload-processing" Aug 01 01:14:04.885198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885136 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f57922bb-a3bb-4783-a4a4-a86a54e419b4" containerName="payload-processing" Aug 01 01:14:04.885198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885144 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec23a91f-b0c7-4211-801f-571530f27a61" containerName="payload-processing" Aug 01 01:14:04.885198 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885149 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec23a91f-b0c7-4211-801f-571530f27a61" containerName="payload-processing" Aug 01 01:14:04.885639 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885216 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="794e461d-ac8d-49d7-b4ab-13563d90ca94" containerName="payload-pre-processing" Aug 01 01:14:04.885639 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885226 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec23a91f-b0c7-4211-801f-571530f27a61" containerName="payload-processing" Aug 01 01:14:04.885639 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885237 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" Aug 01 01:14:04.885639 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885247 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c4731a-1109-4eea-8300-2a38127ae1ff" containerName="istio-proxy" Aug 01 01:14:04.885639 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.885265 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f57922bb-a3bb-4783-a4a4-a86a54e419b4" containerName="payload-processing" Aug 01 01:14:04.889840 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.889822 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:04.892461 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.892438 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-1e2f5b-openshift-default-dockercfg-9sl9g\"" Aug 01 01:14:04.899118 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.899096 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8"] Aug 01 01:14:04.970300 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.970262 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-workload-socket\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:04.970503 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.970322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-envoy\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:04.970503 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.970405 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwpfw\" (UniqueName: \"kubernetes.io/projected/9558c6fe-3d0a-4915-ae05-7d6701b3891d-kube-api-access-nwpfw\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:04.970503 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.970460 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-workload-certs\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:04.970503 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.970494 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-data\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:04.970725 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.970521 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-credential-socket\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:04.970725 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.970647 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istiod-ca-cert\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:04.970725 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.970686 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-podinfo\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:04.970725 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:04.970712 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-token\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072066 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072008 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istiod-ca-cert\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072532 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072083 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-podinfo\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072532 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072101 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-token\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072532 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072137 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-workload-socket\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072532 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072176 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-envoy\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072532 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072209 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwpfw\" (UniqueName: \"kubernetes.io/projected/9558c6fe-3d0a-4915-ae05-7d6701b3891d-kube-api-access-nwpfw\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072532 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072248 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-workload-certs\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072532 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072280 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-data\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072532 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072308 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-credential-socket\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072943 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072548 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-workload-socket\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072943 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072793 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-credential-socket\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072943 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072814 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-workload-certs\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072943 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072845 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istiod-ca-cert\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.072943 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.072850 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-data\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.074435 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.074411 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-envoy\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.074653 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.074633 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-podinfo\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.083547 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.083528 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-token\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.083711 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.083695 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwpfw\" (UniqueName: \"kubernetes.io/projected/9558c6fe-3d0a-4915-ae05-7d6701b3891d-kube-api-access-nwpfw\") pod \"e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.200805 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.200733 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:05.335993 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.335968 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8"] Aug 01 01:14:05.338173 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:14:05.338149 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558c6fe_3d0a_4915_ae05_7d6701b3891d.slice/crio-718da4a59d316b37ed5986ceace9071f1bedd0d72ec32aa4a7af0851b019b62e WatchSource:0}: Error finding container 718da4a59d316b37ed5986ceace9071f1bedd0d72ec32aa4a7af0851b019b62e: Status 404 returned error can't find the container with id 718da4a59d316b37ed5986ceace9071f1bedd0d72ec32aa4a7af0851b019b62e Aug 01 01:14:05.340355 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.340317 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:14:05.340454 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.340393 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:14:05.340454 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.340428 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:14:05.669353 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.669314 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" event={"ID":"9558c6fe-3d0a-4915-ae05-7d6701b3891d","Type":"ContainerStarted","Data":"9d7c8e097422b1abfbaa81f13de48a160b7414dd1c757d37181210aa52da6eba"} Aug 01 01:14:05.669531 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.669367 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" event={"ID":"9558c6fe-3d0a-4915-ae05-7d6701b3891d","Type":"ContainerStarted","Data":"718da4a59d316b37ed5986ceace9071f1bedd0d72ec32aa4a7af0851b019b62e"} Aug 01 01:14:05.689376 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:05.689331 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" podStartSLOduration=1.6893153779999999 podStartE2EDuration="1.689315378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:14:05.68767157 +0000 UTC m=+2734.476393409" watchObservedRunningTime="2026-08-01 01:14:05.689315378 +0000 UTC m=+2734.478037204" Aug 01 01:14:06.201994 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:06.201960 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:06.207253 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:06.207228 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:06.674456 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:06.674423 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:06.675436 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:06.675418 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:09.297310 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.297285 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:14:09.389018 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.388987 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw"] Aug 01 01:14:09.389383 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.389368 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerName="istio-proxy" Aug 01 01:14:09.389432 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.389387 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerName="istio-proxy" Aug 01 01:14:09.389432 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.389401 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerName="istio-proxy" Aug 01 01:14:09.389432 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.389410 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerName="istio-proxy" Aug 01 01:14:09.389533 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.389520 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerName="istio-proxy" Aug 01 01:14:09.392425 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.392408 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" Aug 01 01:14:09.395073 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.395045 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-1e2f5b\"" Aug 01 01:14:09.403175 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.403153 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw"] Aug 01 01:14:09.416960 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.416938 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c5gd\" (UniqueName: \"kubernetes.io/projected/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-kube-api-access-5c5gd\") pod \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " Aug 01 01:14:09.417596 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.416985 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-envoy\") pod \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " Aug 01 01:14:09.417596 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.417012 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-credential-socket\") pod \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " Aug 01 01:14:09.417596 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.417091 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-data\") pod \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " Aug 01 01:14:09.417596 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.417118 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-podinfo\") pod \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " Aug 01 01:14:09.417596 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.417157 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-workload-socket\") pod \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " Aug 01 01:14:09.417596 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.417202 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-workload-certs\") pod \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " Aug 01 01:14:09.417596 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.417256 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-token\") pod \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " Aug 01 01:14:09.417596 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.417290 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istiod-ca-cert\") pod \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\" (UID: \"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426\") " Aug 01 01:14:09.418047 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.417910 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" (UID: "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:14:09.418047 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.417972 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" (UID: "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:14:09.418471 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.418184 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" (UID: "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:14:09.418471 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.418254 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-data" (OuterVolumeSpecName: "istio-data") pod "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" (UID: "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:14:09.418471 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.418436 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" (UID: "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:14:09.420132 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.420105 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-kube-api-access-5c5gd" (OuterVolumeSpecName: "kube-api-access-5c5gd") pod "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" (UID: "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426"). InnerVolumeSpecName "kube-api-access-5c5gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:14:09.420580 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.420552 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" (UID: "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:14:09.420580 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.420557 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" (UID: "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 01 01:14:09.420814 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.420796 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-token" (OuterVolumeSpecName: "istio-token") pod "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" (UID: "5db2f9d0-627e-4dd4-8b5f-f8455c8a0426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:14:09.427638 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.427616 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759"] Aug 01 01:14:09.428071 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.428045 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerName="istio-proxy" Aug 01 01:14:09.431549 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.431531 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" Aug 01 01:14:09.436758 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.436737 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-1e2f5b-dockercfg-mn74c\"" Aug 01 01:14:09.448729 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.448661 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759"] Aug 01 01:14:09.519914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.518687 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4ceb57-ddbe-484e-8812-a9b00ea82690-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw\" (UID: \"4f4ceb57-ddbe-484e-8812-a9b00ea82690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" Aug 01 01:14:09.519914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.518764 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvdlk\" (UniqueName: \"kubernetes.io/projected/56d5fa20-5cb0-4272-a589-5d1ee98a476d-kube-api-access-tvdlk\") pod \"payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759\" (UID: \"56d5fa20-5cb0-4272-a589-5d1ee98a476d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" Aug 01 01:14:09.519914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.518914 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kblt2\" (UniqueName: \"kubernetes.io/projected/4f4ceb57-ddbe-484e-8812-a9b00ea82690-kube-api-access-kblt2\") pod \"payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw\" (UID: \"4f4ceb57-ddbe-484e-8812-a9b00ea82690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" Aug 01 01:14:09.519914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.518979 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d5fa20-5cb0-4272-a589-5d1ee98a476d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759\" (UID: \"56d5fa20-5cb0-4272-a589-5d1ee98a476d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" Aug 01 01:14:09.519914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.519110 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-workload-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:14:09.519914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.519130 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-workload-certs\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:14:09.519914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.519141 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-token\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:14:09.519914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.519148 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istiod-ca-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:14:09.519914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.519159 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c5gd\" (UniqueName: \"kubernetes.io/projected/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-kube-api-access-5c5gd\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:14:09.519914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.519172 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-envoy\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:14:09.519914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.519185 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-credential-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:14:09.519914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.519197 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-data\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:14:09.519914 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.519205 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426-istio-podinfo\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:14:09.620640 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.620606 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvdlk\" (UniqueName: \"kubernetes.io/projected/56d5fa20-5cb0-4272-a589-5d1ee98a476d-kube-api-access-tvdlk\") pod \"payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759\" (UID: \"56d5fa20-5cb0-4272-a589-5d1ee98a476d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" Aug 01 01:14:09.620835 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.620666 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kblt2\" (UniqueName: \"kubernetes.io/projected/4f4ceb57-ddbe-484e-8812-a9b00ea82690-kube-api-access-kblt2\") pod \"payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw\" (UID: \"4f4ceb57-ddbe-484e-8812-a9b00ea82690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" Aug 01 01:14:09.620835 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.620700 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d5fa20-5cb0-4272-a589-5d1ee98a476d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759\" (UID: \"56d5fa20-5cb0-4272-a589-5d1ee98a476d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" Aug 01 01:14:09.620835 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.620745 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4ceb57-ddbe-484e-8812-a9b00ea82690-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw\" (UID: \"4f4ceb57-ddbe-484e-8812-a9b00ea82690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" Aug 01 01:14:09.621508 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.621483 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d5fa20-5cb0-4272-a589-5d1ee98a476d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759\" (UID: \"56d5fa20-5cb0-4272-a589-5d1ee98a476d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" Aug 01 01:14:09.621607 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.621516 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4ceb57-ddbe-484e-8812-a9b00ea82690-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw\" (UID: \"4f4ceb57-ddbe-484e-8812-a9b00ea82690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" Aug 01 01:14:09.630983 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.630950 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvdlk\" (UniqueName: \"kubernetes.io/projected/56d5fa20-5cb0-4272-a589-5d1ee98a476d-kube-api-access-tvdlk\") pod \"payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759\" (UID: \"56d5fa20-5cb0-4272-a589-5d1ee98a476d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" Aug 01 01:14:09.631895 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.631864 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kblt2\" (UniqueName: \"kubernetes.io/projected/4f4ceb57-ddbe-484e-8812-a9b00ea82690-kube-api-access-kblt2\") pod \"payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw\" (UID: \"4f4ceb57-ddbe-484e-8812-a9b00ea82690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" Aug 01 01:14:09.690019 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.689952 2563 generic.go:356] "Generic (PLEG): container finished" podID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" containerID="ac8f9e531c6ce357b78f40d6c062015358456166022330bd7725c026a9ffcf24" exitCode=0 Aug 01 01:14:09.690019 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.689990 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" event={"ID":"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426","Type":"ContainerDied","Data":"ac8f9e531c6ce357b78f40d6c062015358456166022330bd7725c026a9ffcf24"} Aug 01 01:14:09.690019 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.690007 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" Aug 01 01:14:09.690262 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.690032 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m" event={"ID":"5db2f9d0-627e-4dd4-8b5f-f8455c8a0426","Type":"ContainerDied","Data":"9392cd73c4fa63a4be9dfc62d9e7bab8a5623d81a2bec5223f9b68ec83e69c46"} Aug 01 01:14:09.690262 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.690084 2563 scope.go:117] "RemoveContainer" containerID="ac8f9e531c6ce357b78f40d6c062015358456166022330bd7725c026a9ffcf24" Aug 01 01:14:09.698846 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.698831 2563 scope.go:117] "RemoveContainer" containerID="c695789161b0ffd7fc9be8f6bb8152b21d53343a992263bb5bd73a65a7212625" Aug 01 01:14:09.704844 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.704827 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" Aug 01 01:14:09.706095 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.706071 2563 scope.go:117] "RemoveContainer" containerID="ac8f9e531c6ce357b78f40d6c062015358456166022330bd7725c026a9ffcf24" Aug 01 01:14:09.706345 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:14:09.706326 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac8f9e531c6ce357b78f40d6c062015358456166022330bd7725c026a9ffcf24\": container with ID starting with ac8f9e531c6ce357b78f40d6c062015358456166022330bd7725c026a9ffcf24 not found: ID does not exist" containerID="ac8f9e531c6ce357b78f40d6c062015358456166022330bd7725c026a9ffcf24" Aug 01 01:14:09.706424 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.706354 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac8f9e531c6ce357b78f40d6c062015358456166022330bd7725c026a9ffcf24"} err="failed to get container status \"ac8f9e531c6ce357b78f40d6c062015358456166022330bd7725c026a9ffcf24\": rpc error: code = NotFound desc = could not find container \"ac8f9e531c6ce357b78f40d6c062015358456166022330bd7725c026a9ffcf24\": container with ID starting with ac8f9e531c6ce357b78f40d6c062015358456166022330bd7725c026a9ffcf24 not found: ID does not exist" Aug 01 01:14:09.706424 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.706379 2563 scope.go:117] "RemoveContainer" containerID="c695789161b0ffd7fc9be8f6bb8152b21d53343a992263bb5bd73a65a7212625" Aug 01 01:14:09.706656 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:14:09.706613 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c695789161b0ffd7fc9be8f6bb8152b21d53343a992263bb5bd73a65a7212625\": container with ID starting with c695789161b0ffd7fc9be8f6bb8152b21d53343a992263bb5bd73a65a7212625 not found: ID does not exist" containerID="c695789161b0ffd7fc9be8f6bb8152b21d53343a992263bb5bd73a65a7212625" Aug 01 01:14:09.706656 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.706637 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c695789161b0ffd7fc9be8f6bb8152b21d53343a992263bb5bd73a65a7212625"} err="failed to get container status \"c695789161b0ffd7fc9be8f6bb8152b21d53343a992263bb5bd73a65a7212625\": rpc error: code = NotFound desc = could not find container \"c695789161b0ffd7fc9be8f6bb8152b21d53343a992263bb5bd73a65a7212625\": container with ID starting with c695789161b0ffd7fc9be8f6bb8152b21d53343a992263bb5bd73a65a7212625 not found: ID does not exist" Aug 01 01:14:09.730918 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.730887 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m"] Aug 01 01:14:09.737551 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.737529 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-113a68-openshift-default-6dd8f6d5f4-5t89m"] Aug 01 01:14:09.754483 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.754460 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" Aug 01 01:14:09.908203 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:09.908179 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759"] Aug 01 01:14:09.910149 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:14:09.910117 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d5fa20_5cb0_4272_a589_5d1ee98a476d.slice/crio-4bb4972275bcf79de8d7ab890c015b0eed2f97904b037233fe322224b84510ea WatchSource:0}: Error finding container 4bb4972275bcf79de8d7ab890c015b0eed2f97904b037233fe322224b84510ea: Status 404 returned error can't find the container with id 4bb4972275bcf79de8d7ab890c015b0eed2f97904b037233fe322224b84510ea Aug 01 01:14:10.051160 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:10.051071 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw"] Aug 01 01:14:10.053700 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:14:10.053669 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4ceb57_ddbe_484e_8812_a9b00ea82690.slice/crio-ca83157e1f865f29e8ef2898504ac19e68fc60ddf3a3f47e5dba87015cd1eb28 WatchSource:0}: Error finding container ca83157e1f865f29e8ef2898504ac19e68fc60ddf3a3f47e5dba87015cd1eb28: Status 404 returned error can't find the container with id ca83157e1f865f29e8ef2898504ac19e68fc60ddf3a3f47e5dba87015cd1eb28 Aug 01 01:14:10.696255 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:10.696220 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" event={"ID":"56d5fa20-5cb0-4272-a589-5d1ee98a476d","Type":"ContainerStarted","Data":"f65a524646a7b3eef6f03b09e272d2b4c5c81c974c9a95b3ad973ead3af7871e"} Aug 01 01:14:10.696689 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:10.696272 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" event={"ID":"56d5fa20-5cb0-4272-a589-5d1ee98a476d","Type":"ContainerStarted","Data":"4bb4972275bcf79de8d7ab890c015b0eed2f97904b037233fe322224b84510ea"} Aug 01 01:14:10.696689 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:10.696314 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" Aug 01 01:14:10.698602 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:10.698579 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" event={"ID":"4f4ceb57-ddbe-484e-8812-a9b00ea82690","Type":"ContainerStarted","Data":"c675d3303420d2ca4cd858dd054e7e8425f2a59781c7239a408f2372a34f736a"} Aug 01 01:14:10.698726 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:10.698616 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" event={"ID":"4f4ceb57-ddbe-484e-8812-a9b00ea82690","Type":"ContainerStarted","Data":"ca83157e1f865f29e8ef2898504ac19e68fc60ddf3a3f47e5dba87015cd1eb28"} Aug 01 01:14:10.698726 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:10.698666 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" Aug 01 01:14:10.713966 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:10.713916 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" podStartSLOduration=1.713898329 podStartE2EDuration="1.713898329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:14:10.711690627 +0000 UTC m=+2739.500412456" watchObservedRunningTime="2026-08-01 01:14:10.713898329 +0000 UTC m=+2739.502620157" Aug 01 01:14:10.727544 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:10.727507 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" podStartSLOduration=1.727494837 podStartE2EDuration="1.727494837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:14:10.726205883 +0000 UTC m=+2739.514927709" watchObservedRunningTime="2026-08-01 01:14:10.727494837 +0000 UTC m=+2739.516216663" Aug 01 01:14:11.728230 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:11.728195 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5db2f9d0-627e-4dd4-8b5f-f8455c8a0426" path="/var/lib/kubelet/pods/5db2f9d0-627e-4dd4-8b5f-f8455c8a0426/volumes" Aug 01 01:14:21.676519 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:21.676482 2563 patch_prober.go:28] interesting pod/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Aug 01 01:14:21.676925 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:21.676542 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" podUID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Aug 01 01:14:21.704590 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:21.704552 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" podUID="4f4ceb57-ddbe-484e-8812-a9b00ea82690" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.64:9004: connect: connection refused" Aug 01 01:14:21.705072 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:21.705036 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" Aug 01 01:14:21.766167 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:21.766138 2563 generic.go:356] "Generic (PLEG): container finished" podID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" containerID="9d7c8e097422b1abfbaa81f13de48a160b7414dd1c757d37181210aa52da6eba" exitCode=0 Aug 01 01:14:21.766322 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:21.766207 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" event={"ID":"9558c6fe-3d0a-4915-ae05-7d6701b3891d","Type":"ContainerDied","Data":"9d7c8e097422b1abfbaa81f13de48a160b7414dd1c757d37181210aa52da6eba"} Aug 01 01:14:21.766619 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:21.766606 2563 scope.go:117] "RemoveContainer" containerID="9d7c8e097422b1abfbaa81f13de48a160b7414dd1c757d37181210aa52da6eba" Aug 01 01:14:21.767636 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:21.767610 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:14:21.767726 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:21.767685 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:14:21.767726 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:21.767719 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:14:22.772660 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:22.772627 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" event={"ID":"9558c6fe-3d0a-4915-ae05-7d6701b3891d","Type":"ContainerStarted","Data":"46a5d98156a1c1d8449419afecfffcdae0072b2e9305fa11f9eb83d0de02fd3a"} Aug 01 01:14:23.201721 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:23.201679 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:23.201918 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:23.201896 2563 patch_prober.go:28] interesting pod/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Aug 01 01:14:23.201965 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:23.201942 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" podUID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Aug 01 01:14:24.201733 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:24.201691 2563 patch_prober.go:28] interesting pod/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Aug 01 01:14:24.202141 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:24.201753 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" podUID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Aug 01 01:14:25.201311 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:25.201275 2563 patch_prober.go:28] interesting pod/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Aug 01 01:14:25.201489 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:25.201363 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" podUID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Aug 01 01:14:26.201658 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:26.201624 2563 patch_prober.go:28] interesting pod/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Aug 01 01:14:26.202037 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:26.201680 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" podUID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Aug 01 01:14:27.201614 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:27.201578 2563 patch_prober.go:28] interesting pod/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Aug 01 01:14:27.201793 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:27.201633 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" podUID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Aug 01 01:14:28.205290 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:28.205260 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:28.206664 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:28.206633 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:28.212567 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:28.212544 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:14:31.703955 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:31.703925 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" Aug 01 01:14:33.113441 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.113409 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk"] Aug 01 01:14:33.118768 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.118751 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.122794 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.122767 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-4224e1-openshift-default-dockercfg-kkc9l\"" Aug 01 01:14:33.209675 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.209643 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk"] Aug 01 01:14:33.243824 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.243793 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/042f0d0f-3699-44ae-bbfb-e4f842739398-istiod-ca-cert\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.244031 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.243833 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-workload-certs\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.244031 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.243860 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-workload-socket\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.244031 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.243884 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-data\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.244031 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.243938 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-credential-socket\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.244031 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.244012 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-token\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.244307 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.244040 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-envoy\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.244307 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.244131 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-podinfo\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.244307 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.244185 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbdd2\" (UniqueName: \"kubernetes.io/projected/042f0d0f-3699-44ae-bbfb-e4f842739398-kube-api-access-cbdd2\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.345005 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.344972 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbdd2\" (UniqueName: \"kubernetes.io/projected/042f0d0f-3699-44ae-bbfb-e4f842739398-kube-api-access-cbdd2\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.345220 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.345029 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/042f0d0f-3699-44ae-bbfb-e4f842739398-istiod-ca-cert\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.345220 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.345087 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-workload-certs\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.345220 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.345208 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-workload-socket\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.345392 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.345250 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-data\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.345392 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.345269 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-credential-socket\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.345392 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.345342 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-token\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.345550 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.345434 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-envoy\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.345550 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.345497 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-podinfo\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.345550 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.345498 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-workload-certs\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.345719 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.345698 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-workload-socket\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.345780 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.345727 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-data\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.345833 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.345815 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-credential-socket\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.346145 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.346120 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/042f0d0f-3699-44ae-bbfb-e4f842739398-istiod-ca-cert\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.347714 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.347691 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-envoy\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.347962 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.347944 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-podinfo\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.353461 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.353432 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-token\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.353651 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.353631 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbdd2\" (UniqueName: \"kubernetes.io/projected/042f0d0f-3699-44ae-bbfb-e4f842739398-kube-api-access-cbdd2\") pod \"e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.430444 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.430350 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:33.571536 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.571510 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk"] Aug 01 01:14:33.573312 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:14:33.573286 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042f0d0f_3699_44ae_bbfb_e4f842739398.slice/crio-2395326c734cae09001553892298fe99061824cb3f88cf7ee0bbc947b274f4c4 WatchSource:0}: Error finding container 2395326c734cae09001553892298fe99061824cb3f88cf7ee0bbc947b274f4c4: Status 404 returned error can't find the container with id 2395326c734cae09001553892298fe99061824cb3f88cf7ee0bbc947b274f4c4 Aug 01 01:14:33.575456 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.575423 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:14:33.575540 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.575491 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:14:33.575540 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.575517 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:14:33.831908 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.831867 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" event={"ID":"042f0d0f-3699-44ae-bbfb-e4f842739398","Type":"ContainerStarted","Data":"436fb01adaf18253b0e3b1f21dbb26ecb1f280a9997c426eed698e1f5c226501"} Aug 01 01:14:33.832135 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.831931 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" event={"ID":"042f0d0f-3699-44ae-bbfb-e4f842739398","Type":"ContainerStarted","Data":"2395326c734cae09001553892298fe99061824cb3f88cf7ee0bbc947b274f4c4"} Aug 01 01:14:33.856846 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:33.856789 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" podStartSLOduration=0.856772757 podStartE2EDuration="856.772757ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:14:33.854853559 +0000 UTC m=+2762.643575411" watchObservedRunningTime="2026-08-01 01:14:33.856772757 +0000 UTC m=+2762.645494582" Aug 01 01:14:34.430787 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:34.430755 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:34.436161 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:34.436136 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:34.837557 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:34.837531 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:34.838724 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:34.838700 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:37.488892 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:37.488855 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r"] Aug 01 01:14:37.492478 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:37.492458 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" Aug 01 01:14:37.494994 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:37.494973 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-4224e1-dockercfg-89255\"" Aug 01 01:14:37.494994 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:37.494989 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-4224e1\"" Aug 01 01:14:37.502334 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:37.502313 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r"] Aug 01 01:14:37.684470 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:37.684429 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwgwh\" (UniqueName: \"kubernetes.io/projected/71ab65c0-6274-4804-b03f-7cf02765bddb-kube-api-access-xwgwh\") pod \"payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r\" (UID: \"71ab65c0-6274-4804-b03f-7cf02765bddb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" Aug 01 01:14:37.684658 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:37.684493 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ab65c0-6274-4804-b03f-7cf02765bddb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r\" (UID: \"71ab65c0-6274-4804-b03f-7cf02765bddb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" Aug 01 01:14:37.785609 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:37.785528 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwgwh\" (UniqueName: \"kubernetes.io/projected/71ab65c0-6274-4804-b03f-7cf02765bddb-kube-api-access-xwgwh\") pod \"payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r\" (UID: \"71ab65c0-6274-4804-b03f-7cf02765bddb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" Aug 01 01:14:37.785609 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:37.785581 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ab65c0-6274-4804-b03f-7cf02765bddb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r\" (UID: \"71ab65c0-6274-4804-b03f-7cf02765bddb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" Aug 01 01:14:37.786279 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:37.786259 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ab65c0-6274-4804-b03f-7cf02765bddb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r\" (UID: \"71ab65c0-6274-4804-b03f-7cf02765bddb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" Aug 01 01:14:37.795050 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:37.795023 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwgwh\" (UniqueName: \"kubernetes.io/projected/71ab65c0-6274-4804-b03f-7cf02765bddb-kube-api-access-xwgwh\") pod \"payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r\" (UID: \"71ab65c0-6274-4804-b03f-7cf02765bddb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" Aug 01 01:14:37.804786 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:37.804762 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" Aug 01 01:14:37.941667 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:37.941635 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r"] Aug 01 01:14:37.943051 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:14:37.943025 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ab65c0_6274_4804_b03f_7cf02765bddb.slice/crio-30d414f4a17919017f782672cc26f9ac7b68545dc320ea097b677649f84e867e WatchSource:0}: Error finding container 30d414f4a17919017f782672cc26f9ac7b68545dc320ea097b677649f84e867e: Status 404 returned error can't find the container with id 30d414f4a17919017f782672cc26f9ac7b68545dc320ea097b677649f84e867e Aug 01 01:14:38.860557 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:38.860520 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" event={"ID":"71ab65c0-6274-4804-b03f-7cf02765bddb","Type":"ContainerStarted","Data":"9a8b465ea3694be4867acf34d7770c088a2e9ae00712feb70ee170ab9d527c0e"} Aug 01 01:14:38.860970 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:38.860579 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" event={"ID":"71ab65c0-6274-4804-b03f-7cf02765bddb","Type":"ContainerStarted","Data":"30d414f4a17919017f782672cc26f9ac7b68545dc320ea097b677649f84e867e"} Aug 01 01:14:38.860970 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:38.860621 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" Aug 01 01:14:38.878560 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:38.878512 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" podStartSLOduration=1.878497656 podStartE2EDuration="1.878497656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:14:38.876116654 +0000 UTC m=+2767.664838483" watchObservedRunningTime="2026-08-01 01:14:38.878497656 +0000 UTC m=+2767.667219507" Aug 01 01:14:48.911864 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:48.911785 2563 generic.go:356] "Generic (PLEG): container finished" podID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerID="436fb01adaf18253b0e3b1f21dbb26ecb1f280a9997c426eed698e1f5c226501" exitCode=0 Aug 01 01:14:48.912375 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:48.911861 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" event={"ID":"042f0d0f-3699-44ae-bbfb-e4f842739398","Type":"ContainerDied","Data":"436fb01adaf18253b0e3b1f21dbb26ecb1f280a9997c426eed698e1f5c226501"} Aug 01 01:14:48.912375 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:48.912325 2563 scope.go:117] "RemoveContainer" containerID="436fb01adaf18253b0e3b1f21dbb26ecb1f280a9997c426eed698e1f5c226501" Aug 01 01:14:48.913380 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:48.913350 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:14:48.913476 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:48.913424 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:14:48.913476 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:48.913459 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:14:49.838435 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:49.838397 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:49.867512 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:49.867480 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" Aug 01 01:14:49.935395 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:49.935357 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" event={"ID":"042f0d0f-3699-44ae-bbfb-e4f842739398","Type":"ContainerStarted","Data":"30e6ad3cf141b5d79d325c347c92d6588c5b420a0d68a3582860fee562ffb7c2"} Aug 01 01:14:50.431106 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:50.431068 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:50.432456 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:50.432377 2563 patch_prober.go:28] interesting pod/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Aug 01 01:14:50.432651 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:50.432628 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" podUID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Aug 01 01:14:51.430822 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:51.430782 2563 patch_prober.go:28] interesting pod/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Aug 01 01:14:51.431503 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:51.431469 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" podUID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Aug 01 01:14:52.437348 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:52.437298 2563 patch_prober.go:28] interesting pod/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Aug 01 01:14:52.437865 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:52.437357 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" podUID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Aug 01 01:14:53.431022 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:53.430981 2563 patch_prober.go:28] interesting pod/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Aug 01 01:14:53.431348 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:53.431319 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" podUID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Aug 01 01:14:54.431212 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:54.431125 2563 patch_prober.go:28] interesting pod/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Aug 01 01:14:54.431973 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:54.431945 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" podUID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Aug 01 01:14:55.434392 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:55.434360 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:55.434759 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:55.434726 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:14:55.435455 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:14:55.435436 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:18:31.908101 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:18:31.907999 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 01:18:31.913882 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:18:31.913862 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 01:18:31.916403 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:18:31.916381 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 01:18:31.921958 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:18:31.921936 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 01:22:38.903046 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:38.902960 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv"] Aug 01 01:22:38.906830 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:38.906808 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:38.909758 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:38.909736 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2db7f1-openshift-default-dockercfg-nhkk4\"" Aug 01 01:22:38.927141 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:38.927120 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv"] Aug 01 01:22:39.026764 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.026732 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xpc8\" (UniqueName: \"kubernetes.io/projected/9de5ae02-8385-40fe-8803-08452b9fd73b-kube-api-access-5xpc8\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.026934 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.026769 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-envoy\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.026934 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.026891 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-credential-socket\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.027032 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.026937 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de5ae02-8385-40fe-8803-08452b9fd73b-istiod-ca-cert\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.027032 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.026989 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-workload-certs\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.027032 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.027017 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-podinfo\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.027190 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.027040 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-token\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.027190 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.027130 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-workload-socket\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.027269 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.027196 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-data\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.128709 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.128671 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de5ae02-8385-40fe-8803-08452b9fd73b-istiod-ca-cert\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.128888 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.128722 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-workload-certs\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.128888 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.128745 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-podinfo\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.128888 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.128760 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-token\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.128888 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.128786 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-workload-socket\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.128888 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.128817 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-data\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.129220 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.128897 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpc8\" (UniqueName: \"kubernetes.io/projected/9de5ae02-8385-40fe-8803-08452b9fd73b-kube-api-access-5xpc8\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.129220 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.128924 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-envoy\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.129220 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.128971 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-credential-socket\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.129379 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.129299 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-workload-socket\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.129490 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.129466 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-credential-socket\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.129554 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.129477 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-data\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.129610 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.129570 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de5ae02-8385-40fe-8803-08452b9fd73b-istiod-ca-cert\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.129705 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.129686 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-workload-certs\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.131598 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.131576 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-envoy\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.131803 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.131787 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-podinfo\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.136587 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.136565 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-token\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.136773 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.136752 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpc8\" (UniqueName: \"kubernetes.io/projected/9de5ae02-8385-40fe-8803-08452b9fd73b-kube-api-access-5xpc8\") pod \"e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.218152 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.218066 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:39.358448 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.358380 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv"] Aug 01 01:22:39.360752 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:22:39.360725 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de5ae02_8385_40fe_8803_08452b9fd73b.slice/crio-d492477f6be64650ebbf7f479f7157be6dcf31a2ebf9e065659ec6c293f723b7 WatchSource:0}: Error finding container d492477f6be64650ebbf7f479f7157be6dcf31a2ebf9e065659ec6c293f723b7: Status 404 returned error can't find the container with id d492477f6be64650ebbf7f479f7157be6dcf31a2ebf9e065659ec6c293f723b7 Aug 01 01:22:39.362972 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.362924 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:22:39.363051 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.363029 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:22:39.363135 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:39.363116 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:22:40.240392 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:40.240347 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" event={"ID":"9de5ae02-8385-40fe-8803-08452b9fd73b","Type":"ContainerStarted","Data":"f3b4e8af3a015ac398ee2551ba9e63005ebf0831837b2e300b3f426458998a90"} Aug 01 01:22:40.240783 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:40.240405 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" event={"ID":"9de5ae02-8385-40fe-8803-08452b9fd73b","Type":"ContainerStarted","Data":"d492477f6be64650ebbf7f479f7157be6dcf31a2ebf9e065659ec6c293f723b7"} Aug 01 01:22:40.263860 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:40.263767 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" podStartSLOduration=2.263746656 podStartE2EDuration="2.263746656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:22:40.259899857 +0000 UTC m=+3249.048621685" watchObservedRunningTime="2026-08-01 01:22:40.263746656 +0000 UTC m=+3249.052468483" Aug 01 01:22:41.219117 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:41.219074 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:41.224653 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:41.224627 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:41.246152 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:41.246119 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:41.246918 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:41.246901 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:56.406613 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:56.405326 2563 patch_prober.go:28] interesting pod/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": read tcp 10.132.0.2:56238->10.132.0.68:15021: read: connection reset by peer" start-of-body= Aug 01 01:22:56.406613 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:56.405403 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": read tcp 10.132.0.2:56238->10.132.0.68:15021: read: connection reset by peer" Aug 01 01:22:57.330542 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:57.330508 2563 generic.go:356] "Generic (PLEG): container finished" podID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerID="f3b4e8af3a015ac398ee2551ba9e63005ebf0831837b2e300b3f426458998a90" exitCode=0 Aug 01 01:22:57.330718 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:57.330575 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" event={"ID":"9de5ae02-8385-40fe-8803-08452b9fd73b","Type":"ContainerDied","Data":"f3b4e8af3a015ac398ee2551ba9e63005ebf0831837b2e300b3f426458998a90"} Aug 01 01:22:57.330942 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:57.330928 2563 scope.go:117] "RemoveContainer" containerID="f3b4e8af3a015ac398ee2551ba9e63005ebf0831837b2e300b3f426458998a90" Aug 01 01:22:57.331942 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:57.331915 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:22:57.331999 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:57.331986 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:22:57.332037 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:57.332026 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Aug 01 01:22:58.340425 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:58.340391 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" event={"ID":"9de5ae02-8385-40fe-8803-08452b9fd73b","Type":"ContainerStarted","Data":"24e727a0d3261a1a7a4c26e9b97e4498237b254b86b6b89be9f91be1625026ec"} Aug 01 01:22:59.218605 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:59.218573 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:22:59.218982 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:59.218958 2563 patch_prober.go:28] interesting pod/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Aug 01 01:22:59.219195 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:22:59.219165 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Aug 01 01:23:00.222434 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:00.222400 2563 patch_prober.go:28] interesting pod/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Aug 01 01:23:00.222998 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:00.222967 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Aug 01 01:23:01.219840 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:01.219425 2563 patch_prober.go:28] interesting pod/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Aug 01 01:23:01.219840 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:01.219484 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Aug 01 01:23:02.219043 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:02.219004 2563 patch_prober.go:28] interesting pod/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Aug 01 01:23:02.219619 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:02.219586 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Aug 01 01:23:03.218997 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:03.218958 2563 patch_prober.go:28] interesting pod/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Aug 01 01:23:03.219311 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:03.219282 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Aug 01 01:23:04.222668 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:04.222638 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:23:04.223049 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:04.222981 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:23:04.223750 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:04.223731 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:23:12.879490 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:12.879457 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg"] Aug 01 01:23:12.883279 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:12.883260 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" Aug 01 01:23:12.885734 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:12.885713 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-576e07\"" Aug 01 01:23:12.893445 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:12.893423 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg"] Aug 01 01:23:12.924004 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:12.923975 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c"] Aug 01 01:23:12.927550 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:12.927532 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" Aug 01 01:23:12.930292 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:12.930277 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-576e07-dockercfg-m4qk9\"" Aug 01 01:23:12.937285 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:12.937266 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c"] Aug 01 01:23:13.061467 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.061418 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382130f6-42bd-45d5-985a-59b9d0b675ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c\" (UID: \"382130f6-42bd-45d5-985a-59b9d0b675ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" Aug 01 01:23:13.061642 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.061492 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qfzg\" (UniqueName: \"kubernetes.io/projected/1b9b0e49-7a01-4c78-b400-24a49158850b-kube-api-access-5qfzg\") pod \"payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg\" (UID: \"1b9b0e49-7a01-4c78-b400-24a49158850b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" Aug 01 01:23:13.061642 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.061550 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9b0e49-7a01-4c78-b400-24a49158850b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg\" (UID: \"1b9b0e49-7a01-4c78-b400-24a49158850b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" Aug 01 01:23:13.061642 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.061567 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6z4\" (UniqueName: \"kubernetes.io/projected/382130f6-42bd-45d5-985a-59b9d0b675ff-kube-api-access-kr6z4\") pod \"payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c\" (UID: \"382130f6-42bd-45d5-985a-59b9d0b675ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" Aug 01 01:23:13.162503 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.162425 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9b0e49-7a01-4c78-b400-24a49158850b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg\" (UID: \"1b9b0e49-7a01-4c78-b400-24a49158850b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" Aug 01 01:23:13.162503 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.162459 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6z4\" (UniqueName: \"kubernetes.io/projected/382130f6-42bd-45d5-985a-59b9d0b675ff-kube-api-access-kr6z4\") pod \"payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c\" (UID: \"382130f6-42bd-45d5-985a-59b9d0b675ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" Aug 01 01:23:13.162503 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.162500 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382130f6-42bd-45d5-985a-59b9d0b675ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c\" (UID: \"382130f6-42bd-45d5-985a-59b9d0b675ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" Aug 01 01:23:13.162763 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.162538 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qfzg\" (UniqueName: \"kubernetes.io/projected/1b9b0e49-7a01-4c78-b400-24a49158850b-kube-api-access-5qfzg\") pod \"payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg\" (UID: \"1b9b0e49-7a01-4c78-b400-24a49158850b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" Aug 01 01:23:13.163267 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.163241 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9b0e49-7a01-4c78-b400-24a49158850b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg\" (UID: \"1b9b0e49-7a01-4c78-b400-24a49158850b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" Aug 01 01:23:13.163365 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.163268 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382130f6-42bd-45d5-985a-59b9d0b675ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c\" (UID: \"382130f6-42bd-45d5-985a-59b9d0b675ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" Aug 01 01:23:13.170265 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.170241 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6z4\" (UniqueName: \"kubernetes.io/projected/382130f6-42bd-45d5-985a-59b9d0b675ff-kube-api-access-kr6z4\") pod \"payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c\" (UID: \"382130f6-42bd-45d5-985a-59b9d0b675ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" Aug 01 01:23:13.170375 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.170325 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qfzg\" (UniqueName: \"kubernetes.io/projected/1b9b0e49-7a01-4c78-b400-24a49158850b-kube-api-access-5qfzg\") pod \"payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg\" (UID: \"1b9b0e49-7a01-4c78-b400-24a49158850b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" Aug 01 01:23:13.194235 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.194213 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" Aug 01 01:23:13.238178 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.238144 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" Aug 01 01:23:13.344923 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.344897 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg"] Aug 01 01:23:13.347654 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:23:13.347628 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9b0e49_7a01_4c78_b400_24a49158850b.slice/crio-5f54ed31f81f5850e54bed490d593be108a6742fca030ba193cba044fc38e795 WatchSource:0}: Error finding container 5f54ed31f81f5850e54bed490d593be108a6742fca030ba193cba044fc38e795: Status 404 returned error can't find the container with id 5f54ed31f81f5850e54bed490d593be108a6742fca030ba193cba044fc38e795 Aug 01 01:23:13.405035 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.405007 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c"] Aug 01 01:23:13.407610 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:23:13.407583 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382130f6_42bd_45d5_985a_59b9d0b675ff.slice/crio-edb5b634fa401e4eab802a3522386265cc657732fdebc8161e967990681a6538 WatchSource:0}: Error finding container edb5b634fa401e4eab802a3522386265cc657732fdebc8161e967990681a6538: Status 404 returned error can't find the container with id edb5b634fa401e4eab802a3522386265cc657732fdebc8161e967990681a6538 Aug 01 01:23:13.420303 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.420275 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" event={"ID":"1b9b0e49-7a01-4c78-b400-24a49158850b","Type":"ContainerStarted","Data":"5f54ed31f81f5850e54bed490d593be108a6742fca030ba193cba044fc38e795"} Aug 01 01:23:13.421548 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:13.421527 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" event={"ID":"382130f6-42bd-45d5-985a-59b9d0b675ff","Type":"ContainerStarted","Data":"edb5b634fa401e4eab802a3522386265cc657732fdebc8161e967990681a6538"} Aug 01 01:23:14.429104 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:14.429044 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" event={"ID":"1b9b0e49-7a01-4c78-b400-24a49158850b","Type":"ContainerStarted","Data":"362af1cd20c403341dc49aea3019e9124ac70281f493ae9168393266739f782f"} Aug 01 01:23:14.429558 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:14.429141 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" Aug 01 01:23:14.430987 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:14.430954 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" event={"ID":"382130f6-42bd-45d5-985a-59b9d0b675ff","Type":"ContainerStarted","Data":"3a29b0d8f966fbdb428dd8020b2099d8f115754575ce4192a00154e3793555e1"} Aug 01 01:23:14.431124 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:14.431101 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" Aug 01 01:23:14.445897 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:14.445858 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" podStartSLOduration=2.445845071 podStartE2EDuration="2.445845071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:23:14.443987661 +0000 UTC m=+3283.232709489" watchObservedRunningTime="2026-08-01 01:23:14.445845071 +0000 UTC m=+3283.234566896" Aug 01 01:23:14.460675 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:14.460628 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" podStartSLOduration=2.460610643 podStartE2EDuration="2.460610643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:23:14.458310652 +0000 UTC m=+3283.247032481" watchObservedRunningTime="2026-08-01 01:23:14.460610643 +0000 UTC m=+3283.249332471" Aug 01 01:23:25.437497 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:25.437464 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" Aug 01 01:23:25.438015 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:25.437660 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" Aug 01 01:23:31.940519 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:31.940253 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 01:23:31.947105 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:31.947076 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 01:23:31.950763 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:31.950742 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 01:23:31.962598 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:31.962571 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b54j8_e7e36f0b-4b54-4945-be93-63ac46767270/ovn-acl-logging/0.log" Aug 01 01:23:43.018427 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:23:43.018390 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c_382130f6-42bd-45d5-985a-59b9d0b675ff/payload-processing/0.log" Aug 01 01:24:25.958641 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:25.958606 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c_382130f6-42bd-45d5-985a-59b9d0b675ff/payload-processing/0.log" Aug 01 01:24:31.380448 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.380415 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr"] Aug 01 01:24:31.383992 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.383975 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" Aug 01 01:24:31.386769 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.386747 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-659af5-dockercfg-xtjcm\"" Aug 01 01:24:31.386769 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.386763 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-659af5\"" Aug 01 01:24:31.395699 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.395678 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr"] Aug 01 01:24:31.411556 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.411534 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8"] Aug 01 01:24:31.415107 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.415089 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" Aug 01 01:24:31.428459 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.428431 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8"] Aug 01 01:24:31.448609 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.448583 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e690ac14-05c4-4818-a694-3b395d0cf84c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr\" (UID: \"e690ac14-05c4-4818-a694-3b395d0cf84c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" Aug 01 01:24:31.448723 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.448682 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv4tw\" (UniqueName: \"kubernetes.io/projected/e690ac14-05c4-4818-a694-3b395d0cf84c-kube-api-access-xv4tw\") pod \"payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr\" (UID: \"e690ac14-05c4-4818-a694-3b395d0cf84c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" Aug 01 01:24:31.549502 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.549463 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbtcp\" (UniqueName: \"kubernetes.io/projected/e51ab2d9-f4ae-47c1-81de-236e5ec3987d-kube-api-access-sbtcp\") pod \"payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8\" (UID: \"e51ab2d9-f4ae-47c1-81de-236e5ec3987d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" Aug 01 01:24:31.549694 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.549514 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e690ac14-05c4-4818-a694-3b395d0cf84c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr\" (UID: \"e690ac14-05c4-4818-a694-3b395d0cf84c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" Aug 01 01:24:31.549694 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.549619 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51ab2d9-f4ae-47c1-81de-236e5ec3987d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8\" (UID: \"e51ab2d9-f4ae-47c1-81de-236e5ec3987d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" Aug 01 01:24:31.549694 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.549667 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv4tw\" (UniqueName: \"kubernetes.io/projected/e690ac14-05c4-4818-a694-3b395d0cf84c-kube-api-access-xv4tw\") pod \"payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr\" (UID: \"e690ac14-05c4-4818-a694-3b395d0cf84c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" Aug 01 01:24:31.550283 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.550258 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e690ac14-05c4-4818-a694-3b395d0cf84c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr\" (UID: \"e690ac14-05c4-4818-a694-3b395d0cf84c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" Aug 01 01:24:31.560153 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.560132 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv4tw\" (UniqueName: \"kubernetes.io/projected/e690ac14-05c4-4818-a694-3b395d0cf84c-kube-api-access-xv4tw\") pod \"payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr\" (UID: \"e690ac14-05c4-4818-a694-3b395d0cf84c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" Aug 01 01:24:31.650340 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.650269 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbtcp\" (UniqueName: \"kubernetes.io/projected/e51ab2d9-f4ae-47c1-81de-236e5ec3987d-kube-api-access-sbtcp\") pod \"payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8\" (UID: \"e51ab2d9-f4ae-47c1-81de-236e5ec3987d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" Aug 01 01:24:31.650482 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.650355 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51ab2d9-f4ae-47c1-81de-236e5ec3987d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8\" (UID: \"e51ab2d9-f4ae-47c1-81de-236e5ec3987d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" Aug 01 01:24:31.651049 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.651029 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51ab2d9-f4ae-47c1-81de-236e5ec3987d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8\" (UID: \"e51ab2d9-f4ae-47c1-81de-236e5ec3987d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" Aug 01 01:24:31.664516 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.664488 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbtcp\" (UniqueName: \"kubernetes.io/projected/e51ab2d9-f4ae-47c1-81de-236e5ec3987d-kube-api-access-sbtcp\") pod \"payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8\" (UID: \"e51ab2d9-f4ae-47c1-81de-236e5ec3987d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" Aug 01 01:24:31.697966 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.697947 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-659af5-dockercfg-xtjcm\"" Aug 01 01:24:31.706428 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.706411 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" Aug 01 01:24:31.725785 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.725749 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" Aug 01 01:24:31.861916 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:31.861883 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr"] Aug 01 01:24:31.864351 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:24:31.864317 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode690ac14_05c4_4818_a694_3b395d0cf84c.slice/crio-5fc1a4a4e587d2d29cc5b77b77b629c11f97de271d801c6e05a5df3c8abd0de6 WatchSource:0}: Error finding container 5fc1a4a4e587d2d29cc5b77b77b629c11f97de271d801c6e05a5df3c8abd0de6: Status 404 returned error can't find the container with id 5fc1a4a4e587d2d29cc5b77b77b629c11f97de271d801c6e05a5df3c8abd0de6 Aug 01 01:24:32.097727 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:32.097703 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8"] Aug 01 01:24:32.100334 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:24:32.100306 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51ab2d9_f4ae_47c1_81de_236e5ec3987d.slice/crio-44850c97ad10986da19f924a245c1ee0607e01a83f60e0047cf30cbda232db59 WatchSource:0}: Error finding container 44850c97ad10986da19f924a245c1ee0607e01a83f60e0047cf30cbda232db59: Status 404 returned error can't find the container with id 44850c97ad10986da19f924a245c1ee0607e01a83f60e0047cf30cbda232db59 Aug 01 01:24:32.828959 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:32.828921 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" event={"ID":"e690ac14-05c4-4818-a694-3b395d0cf84c","Type":"ContainerStarted","Data":"cc842b11a51157ff07a1c3ea0f0f9eab2e26346eed64cb6f35cf23d4945798d6"} Aug 01 01:24:32.829463 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:32.828987 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" event={"ID":"e690ac14-05c4-4818-a694-3b395d0cf84c","Type":"ContainerStarted","Data":"5fc1a4a4e587d2d29cc5b77b77b629c11f97de271d801c6e05a5df3c8abd0de6"} Aug 01 01:24:32.829463 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:32.829021 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" Aug 01 01:24:32.830661 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:32.830639 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" event={"ID":"e51ab2d9-f4ae-47c1-81de-236e5ec3987d","Type":"ContainerStarted","Data":"8b1a49b16c791281b10963e866528a3ab2dfa6443b8efaec69348c55e9971aaa"} Aug 01 01:24:32.830759 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:32.830683 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" event={"ID":"e51ab2d9-f4ae-47c1-81de-236e5ec3987d","Type":"ContainerStarted","Data":"44850c97ad10986da19f924a245c1ee0607e01a83f60e0047cf30cbda232db59"} Aug 01 01:24:32.830759 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:32.830708 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" Aug 01 01:24:32.845348 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:32.845308 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" podStartSLOduration=1.845296003 podStartE2EDuration="1.845296003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:24:32.844525323 +0000 UTC m=+3361.633247176" watchObservedRunningTime="2026-08-01 01:24:32.845296003 +0000 UTC m=+3361.634017886" Aug 01 01:24:32.862873 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:32.862811 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" podStartSLOduration=1.862793355 podStartE2EDuration="1.862793355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:24:32.859329439 +0000 UTC m=+3361.648051266" watchObservedRunningTime="2026-08-01 01:24:32.862793355 +0000 UTC m=+3361.651515183" Aug 01 01:24:43.837912 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:43.837882 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" Aug 01 01:24:43.838390 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:43.837948 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" Aug 01 01:24:52.593275 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.593239 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8"] Aug 01 01:24:52.593760 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.593508 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" podUID="e51ab2d9-f4ae-47c1-81de-236e5ec3987d" containerName="payload-processing" containerID="cri-o://8b1a49b16c791281b10963e866528a3ab2dfa6443b8efaec69348c55e9971aaa" gracePeriod=30 Aug 01 01:24:52.603999 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.603259 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr"] Aug 01 01:24:52.603999 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.603523 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" podUID="e690ac14-05c4-4818-a694-3b395d0cf84c" containerName="payload-pre-processing" containerID="cri-o://cc842b11a51157ff07a1c3ea0f0f9eab2e26346eed64cb6f35cf23d4945798d6" gracePeriod=30 Aug 01 01:24:52.865389 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.865363 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" Aug 01 01:24:52.898409 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.898386 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" Aug 01 01:24:52.937372 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.937340 2563 generic.go:356] "Generic (PLEG): container finished" podID="e51ab2d9-f4ae-47c1-81de-236e5ec3987d" containerID="8b1a49b16c791281b10963e866528a3ab2dfa6443b8efaec69348c55e9971aaa" exitCode=0 Aug 01 01:24:52.937553 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.937381 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" Aug 01 01:24:52.937553 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.937433 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" event={"ID":"e51ab2d9-f4ae-47c1-81de-236e5ec3987d","Type":"ContainerDied","Data":"8b1a49b16c791281b10963e866528a3ab2dfa6443b8efaec69348c55e9971aaa"} Aug 01 01:24:52.937553 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.937493 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8" event={"ID":"e51ab2d9-f4ae-47c1-81de-236e5ec3987d","Type":"ContainerDied","Data":"44850c97ad10986da19f924a245c1ee0607e01a83f60e0047cf30cbda232db59"} Aug 01 01:24:52.937553 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.937527 2563 scope.go:117] "RemoveContainer" containerID="8b1a49b16c791281b10963e866528a3ab2dfa6443b8efaec69348c55e9971aaa" Aug 01 01:24:52.938958 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.938935 2563 generic.go:356] "Generic (PLEG): container finished" podID="e690ac14-05c4-4818-a694-3b395d0cf84c" containerID="cc842b11a51157ff07a1c3ea0f0f9eab2e26346eed64cb6f35cf23d4945798d6" exitCode=0 Aug 01 01:24:52.939048 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.938969 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" Aug 01 01:24:52.939048 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.938970 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" event={"ID":"e690ac14-05c4-4818-a694-3b395d0cf84c","Type":"ContainerDied","Data":"cc842b11a51157ff07a1c3ea0f0f9eab2e26346eed64cb6f35cf23d4945798d6"} Aug 01 01:24:52.939144 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.939118 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr" event={"ID":"e690ac14-05c4-4818-a694-3b395d0cf84c","Type":"ContainerDied","Data":"5fc1a4a4e587d2d29cc5b77b77b629c11f97de271d801c6e05a5df3c8abd0de6"} Aug 01 01:24:52.946251 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.946236 2563 scope.go:117] "RemoveContainer" containerID="8b1a49b16c791281b10963e866528a3ab2dfa6443b8efaec69348c55e9971aaa" Aug 01 01:24:52.946501 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:24:52.946483 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1a49b16c791281b10963e866528a3ab2dfa6443b8efaec69348c55e9971aaa\": container with ID starting with 8b1a49b16c791281b10963e866528a3ab2dfa6443b8efaec69348c55e9971aaa not found: ID does not exist" containerID="8b1a49b16c791281b10963e866528a3ab2dfa6443b8efaec69348c55e9971aaa" Aug 01 01:24:52.946541 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.946511 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1a49b16c791281b10963e866528a3ab2dfa6443b8efaec69348c55e9971aaa"} err="failed to get container status \"8b1a49b16c791281b10963e866528a3ab2dfa6443b8efaec69348c55e9971aaa\": rpc error: code = NotFound desc = could not find container \"8b1a49b16c791281b10963e866528a3ab2dfa6443b8efaec69348c55e9971aaa\": container with ID starting with 8b1a49b16c791281b10963e866528a3ab2dfa6443b8efaec69348c55e9971aaa not found: ID does not exist" Aug 01 01:24:52.946541 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.946527 2563 scope.go:117] "RemoveContainer" containerID="cc842b11a51157ff07a1c3ea0f0f9eab2e26346eed64cb6f35cf23d4945798d6" Aug 01 01:24:52.954273 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.954255 2563 scope.go:117] "RemoveContainer" containerID="cc842b11a51157ff07a1c3ea0f0f9eab2e26346eed64cb6f35cf23d4945798d6" Aug 01 01:24:52.954522 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:24:52.954502 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc842b11a51157ff07a1c3ea0f0f9eab2e26346eed64cb6f35cf23d4945798d6\": container with ID starting with cc842b11a51157ff07a1c3ea0f0f9eab2e26346eed64cb6f35cf23d4945798d6 not found: ID does not exist" containerID="cc842b11a51157ff07a1c3ea0f0f9eab2e26346eed64cb6f35cf23d4945798d6" Aug 01 01:24:52.954583 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:52.954534 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc842b11a51157ff07a1c3ea0f0f9eab2e26346eed64cb6f35cf23d4945798d6"} err="failed to get container status \"cc842b11a51157ff07a1c3ea0f0f9eab2e26346eed64cb6f35cf23d4945798d6\": rpc error: code = NotFound desc = could not find container \"cc842b11a51157ff07a1c3ea0f0f9eab2e26346eed64cb6f35cf23d4945798d6\": container with ID starting with cc842b11a51157ff07a1c3ea0f0f9eab2e26346eed64cb6f35cf23d4945798d6 not found: ID does not exist" Aug 01 01:24:53.061384 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.061359 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv4tw\" (UniqueName: \"kubernetes.io/projected/e690ac14-05c4-4818-a694-3b395d0cf84c-kube-api-access-xv4tw\") pod \"e690ac14-05c4-4818-a694-3b395d0cf84c\" (UID: \"e690ac14-05c4-4818-a694-3b395d0cf84c\") " Aug 01 01:24:53.061541 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.061402 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbtcp\" (UniqueName: \"kubernetes.io/projected/e51ab2d9-f4ae-47c1-81de-236e5ec3987d-kube-api-access-sbtcp\") pod \"e51ab2d9-f4ae-47c1-81de-236e5ec3987d\" (UID: \"e51ab2d9-f4ae-47c1-81de-236e5ec3987d\") " Aug 01 01:24:53.061541 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.061425 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51ab2d9-f4ae-47c1-81de-236e5ec3987d-plugins-config-volume\") pod \"e51ab2d9-f4ae-47c1-81de-236e5ec3987d\" (UID: \"e51ab2d9-f4ae-47c1-81de-236e5ec3987d\") " Aug 01 01:24:53.061541 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.061449 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e690ac14-05c4-4818-a694-3b395d0cf84c-plugins-config-volume\") pod \"e690ac14-05c4-4818-a694-3b395d0cf84c\" (UID: \"e690ac14-05c4-4818-a694-3b395d0cf84c\") " Aug 01 01:24:53.061838 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.061816 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51ab2d9-f4ae-47c1-81de-236e5ec3987d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e51ab2d9-f4ae-47c1-81de-236e5ec3987d" (UID: "e51ab2d9-f4ae-47c1-81de-236e5ec3987d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:24:53.061926 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.061866 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e690ac14-05c4-4818-a694-3b395d0cf84c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e690ac14-05c4-4818-a694-3b395d0cf84c" (UID: "e690ac14-05c4-4818-a694-3b395d0cf84c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:24:53.063483 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.063459 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e690ac14-05c4-4818-a694-3b395d0cf84c-kube-api-access-xv4tw" (OuterVolumeSpecName: "kube-api-access-xv4tw") pod "e690ac14-05c4-4818-a694-3b395d0cf84c" (UID: "e690ac14-05c4-4818-a694-3b395d0cf84c"). InnerVolumeSpecName "kube-api-access-xv4tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:24:53.063564 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.063511 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51ab2d9-f4ae-47c1-81de-236e5ec3987d-kube-api-access-sbtcp" (OuterVolumeSpecName: "kube-api-access-sbtcp") pod "e51ab2d9-f4ae-47c1-81de-236e5ec3987d" (UID: "e51ab2d9-f4ae-47c1-81de-236e5ec3987d"). InnerVolumeSpecName "kube-api-access-sbtcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:24:53.163008 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.162936 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv4tw\" (UniqueName: \"kubernetes.io/projected/e690ac14-05c4-4818-a694-3b395d0cf84c-kube-api-access-xv4tw\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:24:53.163008 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.162961 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbtcp\" (UniqueName: \"kubernetes.io/projected/e51ab2d9-f4ae-47c1-81de-236e5ec3987d-kube-api-access-sbtcp\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:24:53.163008 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.162970 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51ab2d9-f4ae-47c1-81de-236e5ec3987d-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:24:53.163008 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.162978 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e690ac14-05c4-4818-a694-3b395d0cf84c-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:24:53.268625 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.268600 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr"] Aug 01 01:24:53.270598 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.270578 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-659af5-6c5c54bd4-s9vjr"] Aug 01 01:24:53.282128 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.282099 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8"] Aug 01 01:24:53.286809 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.286786 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-659af5-f67b6d9d4-cxqn8"] Aug 01 01:24:53.729867 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.729833 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e51ab2d9-f4ae-47c1-81de-236e5ec3987d" path="/var/lib/kubelet/pods/e51ab2d9-f4ae-47c1-81de-236e5ec3987d/volumes" Aug 01 01:24:53.730274 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:24:53.730238 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e690ac14-05c4-4818-a694-3b395d0cf84c" path="/var/lib/kubelet/pods/e690ac14-05c4-4818-a694-3b395d0cf84c/volumes" Aug 01 01:26:06.625557 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:06.625520 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv"] Aug 01 01:26:06.626171 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:06.625772 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerName="istio-proxy" containerID="cri-o://24e727a0d3261a1a7a4c26e9b97e4498237b254b86b6b89be9f91be1625026ec" gracePeriod=30 Aug 01 01:26:11.898558 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:11.898500 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:26:12.016966 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.016938 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-podinfo\") pod \"9de5ae02-8385-40fe-8803-08452b9fd73b\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " Aug 01 01:26:12.017264 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017001 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de5ae02-8385-40fe-8803-08452b9fd73b-istiod-ca-cert\") pod \"9de5ae02-8385-40fe-8803-08452b9fd73b\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " Aug 01 01:26:12.017264 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017109 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-workload-certs\") pod \"9de5ae02-8385-40fe-8803-08452b9fd73b\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " Aug 01 01:26:12.017264 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017134 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-token\") pod \"9de5ae02-8385-40fe-8803-08452b9fd73b\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " Aug 01 01:26:12.017264 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017172 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-credential-socket\") pod \"9de5ae02-8385-40fe-8803-08452b9fd73b\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " Aug 01 01:26:12.017264 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017194 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-data\") pod \"9de5ae02-8385-40fe-8803-08452b9fd73b\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " Aug 01 01:26:12.017565 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017328 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-workload-socket\") pod \"9de5ae02-8385-40fe-8803-08452b9fd73b\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " Aug 01 01:26:12.017565 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017366 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xpc8\" (UniqueName: \"kubernetes.io/projected/9de5ae02-8385-40fe-8803-08452b9fd73b-kube-api-access-5xpc8\") pod \"9de5ae02-8385-40fe-8803-08452b9fd73b\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " Aug 01 01:26:12.017565 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017393 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-envoy\") pod \"9de5ae02-8385-40fe-8803-08452b9fd73b\" (UID: \"9de5ae02-8385-40fe-8803-08452b9fd73b\") " Aug 01 01:26:12.017565 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017418 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9de5ae02-8385-40fe-8803-08452b9fd73b" (UID: "9de5ae02-8385-40fe-8803-08452b9fd73b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:26:12.017565 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017435 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9de5ae02-8385-40fe-8803-08452b9fd73b" (UID: "9de5ae02-8385-40fe-8803-08452b9fd73b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:26:12.017565 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017479 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de5ae02-8385-40fe-8803-08452b9fd73b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9de5ae02-8385-40fe-8803-08452b9fd73b" (UID: "9de5ae02-8385-40fe-8803-08452b9fd73b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:26:12.017917 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017610 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9de5ae02-8385-40fe-8803-08452b9fd73b" (UID: "9de5ae02-8385-40fe-8803-08452b9fd73b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:26:12.017917 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017717 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-data" (OuterVolumeSpecName: "istio-data") pod "9de5ae02-8385-40fe-8803-08452b9fd73b" (UID: "9de5ae02-8385-40fe-8803-08452b9fd73b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:26:12.017917 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017793 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-credential-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:12.017917 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017811 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-data\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:12.017917 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017826 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-workload-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:12.017917 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017845 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de5ae02-8385-40fe-8803-08452b9fd73b-istiod-ca-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:12.017917 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.017861 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-workload-certs\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:12.019364 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.019341 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9de5ae02-8385-40fe-8803-08452b9fd73b" (UID: "9de5ae02-8385-40fe-8803-08452b9fd73b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 01 01:26:12.019463 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.019358 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-token" (OuterVolumeSpecName: "istio-token") pod "9de5ae02-8385-40fe-8803-08452b9fd73b" (UID: "9de5ae02-8385-40fe-8803-08452b9fd73b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:26:12.019463 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.019413 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de5ae02-8385-40fe-8803-08452b9fd73b-kube-api-access-5xpc8" (OuterVolumeSpecName: "kube-api-access-5xpc8") pod "9de5ae02-8385-40fe-8803-08452b9fd73b" (UID: "9de5ae02-8385-40fe-8803-08452b9fd73b"). InnerVolumeSpecName "kube-api-access-5xpc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:26:12.019572 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.019533 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9de5ae02-8385-40fe-8803-08452b9fd73b" (UID: "9de5ae02-8385-40fe-8803-08452b9fd73b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:26:12.119260 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.119229 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-token\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:12.119260 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.119255 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xpc8\" (UniqueName: \"kubernetes.io/projected/9de5ae02-8385-40fe-8803-08452b9fd73b-kube-api-access-5xpc8\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:12.119434 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.119268 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-envoy\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:12.119434 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.119280 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de5ae02-8385-40fe-8803-08452b9fd73b-istio-podinfo\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:12.344884 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.344849 2563 generic.go:356] "Generic (PLEG): container finished" podID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerID="24e727a0d3261a1a7a4c26e9b97e4498237b254b86b6b89be9f91be1625026ec" exitCode=0 Aug 01 01:26:12.345054 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.344893 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" Aug 01 01:26:12.345054 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.344910 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" event={"ID":"9de5ae02-8385-40fe-8803-08452b9fd73b","Type":"ContainerDied","Data":"24e727a0d3261a1a7a4c26e9b97e4498237b254b86b6b89be9f91be1625026ec"} Aug 01 01:26:12.345054 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.344953 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv" event={"ID":"9de5ae02-8385-40fe-8803-08452b9fd73b","Type":"ContainerDied","Data":"d492477f6be64650ebbf7f479f7157be6dcf31a2ebf9e065659ec6c293f723b7"} Aug 01 01:26:12.345054 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.344978 2563 scope.go:117] "RemoveContainer" containerID="24e727a0d3261a1a7a4c26e9b97e4498237b254b86b6b89be9f91be1625026ec" Aug 01 01:26:12.354331 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.354313 2563 scope.go:117] "RemoveContainer" containerID="f3b4e8af3a015ac398ee2551ba9e63005ebf0831837b2e300b3f426458998a90" Aug 01 01:26:12.362038 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.362020 2563 scope.go:117] "RemoveContainer" containerID="24e727a0d3261a1a7a4c26e9b97e4498237b254b86b6b89be9f91be1625026ec" Aug 01 01:26:12.362318 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:26:12.362298 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e727a0d3261a1a7a4c26e9b97e4498237b254b86b6b89be9f91be1625026ec\": container with ID starting with 24e727a0d3261a1a7a4c26e9b97e4498237b254b86b6b89be9f91be1625026ec not found: ID does not exist" containerID="24e727a0d3261a1a7a4c26e9b97e4498237b254b86b6b89be9f91be1625026ec" Aug 01 01:26:12.362449 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.362326 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e727a0d3261a1a7a4c26e9b97e4498237b254b86b6b89be9f91be1625026ec"} err="failed to get container status \"24e727a0d3261a1a7a4c26e9b97e4498237b254b86b6b89be9f91be1625026ec\": rpc error: code = NotFound desc = could not find container \"24e727a0d3261a1a7a4c26e9b97e4498237b254b86b6b89be9f91be1625026ec\": container with ID starting with 24e727a0d3261a1a7a4c26e9b97e4498237b254b86b6b89be9f91be1625026ec not found: ID does not exist" Aug 01 01:26:12.362449 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.362342 2563 scope.go:117] "RemoveContainer" containerID="f3b4e8af3a015ac398ee2551ba9e63005ebf0831837b2e300b3f426458998a90" Aug 01 01:26:12.362614 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:26:12.362597 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b4e8af3a015ac398ee2551ba9e63005ebf0831837b2e300b3f426458998a90\": container with ID starting with f3b4e8af3a015ac398ee2551ba9e63005ebf0831837b2e300b3f426458998a90 not found: ID does not exist" containerID="f3b4e8af3a015ac398ee2551ba9e63005ebf0831837b2e300b3f426458998a90" Aug 01 01:26:12.362659 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.362619 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b4e8af3a015ac398ee2551ba9e63005ebf0831837b2e300b3f426458998a90"} err="failed to get container status \"f3b4e8af3a015ac398ee2551ba9e63005ebf0831837b2e300b3f426458998a90\": rpc error: code = NotFound desc = could not find container \"f3b4e8af3a015ac398ee2551ba9e63005ebf0831837b2e300b3f426458998a90\": container with ID starting with f3b4e8af3a015ac398ee2551ba9e63005ebf0831837b2e300b3f426458998a90 not found: ID does not exist" Aug 01 01:26:12.366578 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.366554 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv"] Aug 01 01:26:12.370167 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:12.370147 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2db7f1-openshift-default-8584f9df66-wpjhv"] Aug 01 01:26:13.729005 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:13.728976 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" path="/var/lib/kubelet/pods/9de5ae02-8385-40fe-8803-08452b9fd73b/volumes" Aug 01 01:26:17.539678 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.539643 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c"] Aug 01 01:26:17.540161 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.539887 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" podUID="382130f6-42bd-45d5-985a-59b9d0b675ff" containerName="payload-processing" containerID="cri-o://3a29b0d8f966fbdb428dd8020b2099d8f115754575ce4192a00154e3793555e1" gracePeriod=30 Aug 01 01:26:17.547571 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.547543 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg"] Aug 01 01:26:17.547785 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.547764 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" podUID="1b9b0e49-7a01-4c78-b400-24a49158850b" containerName="payload-pre-processing" containerID="cri-o://362af1cd20c403341dc49aea3019e9124ac70281f493ae9168393266739f782f" gracePeriod=30 Aug 01 01:26:17.841084 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.841045 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" Aug 01 01:26:17.843749 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.843731 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" Aug 01 01:26:17.871150 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.870899 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382130f6-42bd-45d5-985a-59b9d0b675ff-plugins-config-volume\") pod \"382130f6-42bd-45d5-985a-59b9d0b675ff\" (UID: \"382130f6-42bd-45d5-985a-59b9d0b675ff\") " Aug 01 01:26:17.871502 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.871479 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382130f6-42bd-45d5-985a-59b9d0b675ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "382130f6-42bd-45d5-985a-59b9d0b675ff" (UID: "382130f6-42bd-45d5-985a-59b9d0b675ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:26:17.971964 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.971938 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9b0e49-7a01-4c78-b400-24a49158850b-plugins-config-volume\") pod \"1b9b0e49-7a01-4c78-b400-24a49158850b\" (UID: \"1b9b0e49-7a01-4c78-b400-24a49158850b\") " Aug 01 01:26:17.972107 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.972037 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr6z4\" (UniqueName: \"kubernetes.io/projected/382130f6-42bd-45d5-985a-59b9d0b675ff-kube-api-access-kr6z4\") pod \"382130f6-42bd-45d5-985a-59b9d0b675ff\" (UID: \"382130f6-42bd-45d5-985a-59b9d0b675ff\") " Aug 01 01:26:17.972107 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.972089 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qfzg\" (UniqueName: \"kubernetes.io/projected/1b9b0e49-7a01-4c78-b400-24a49158850b-kube-api-access-5qfzg\") pod \"1b9b0e49-7a01-4c78-b400-24a49158850b\" (UID: \"1b9b0e49-7a01-4c78-b400-24a49158850b\") " Aug 01 01:26:17.972303 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.972281 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9b0e49-7a01-4c78-b400-24a49158850b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b9b0e49-7a01-4c78-b400-24a49158850b" (UID: "1b9b0e49-7a01-4c78-b400-24a49158850b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:26:17.972303 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.972301 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382130f6-42bd-45d5-985a-59b9d0b675ff-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:17.974142 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.974116 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9b0e49-7a01-4c78-b400-24a49158850b-kube-api-access-5qfzg" (OuterVolumeSpecName: "kube-api-access-5qfzg") pod "1b9b0e49-7a01-4c78-b400-24a49158850b" (UID: "1b9b0e49-7a01-4c78-b400-24a49158850b"). InnerVolumeSpecName "kube-api-access-5qfzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:26:17.974241 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:17.974151 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382130f6-42bd-45d5-985a-59b9d0b675ff-kube-api-access-kr6z4" (OuterVolumeSpecName: "kube-api-access-kr6z4") pod "382130f6-42bd-45d5-985a-59b9d0b675ff" (UID: "382130f6-42bd-45d5-985a-59b9d0b675ff"). InnerVolumeSpecName "kube-api-access-kr6z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:26:18.073130 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.073007 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qfzg\" (UniqueName: \"kubernetes.io/projected/1b9b0e49-7a01-4c78-b400-24a49158850b-kube-api-access-5qfzg\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:18.073130 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.073047 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9b0e49-7a01-4c78-b400-24a49158850b-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:18.073130 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.073090 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr6z4\" (UniqueName: \"kubernetes.io/projected/382130f6-42bd-45d5-985a-59b9d0b675ff-kube-api-access-kr6z4\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:18.377174 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.377095 2563 generic.go:356] "Generic (PLEG): container finished" podID="1b9b0e49-7a01-4c78-b400-24a49158850b" containerID="362af1cd20c403341dc49aea3019e9124ac70281f493ae9168393266739f782f" exitCode=0 Aug 01 01:26:18.377174 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.377140 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" Aug 01 01:26:18.377432 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.377179 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" event={"ID":"1b9b0e49-7a01-4c78-b400-24a49158850b","Type":"ContainerDied","Data":"362af1cd20c403341dc49aea3019e9124ac70281f493ae9168393266739f782f"} Aug 01 01:26:18.377432 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.377230 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg" event={"ID":"1b9b0e49-7a01-4c78-b400-24a49158850b","Type":"ContainerDied","Data":"5f54ed31f81f5850e54bed490d593be108a6742fca030ba193cba044fc38e795"} Aug 01 01:26:18.377432 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.377253 2563 scope.go:117] "RemoveContainer" containerID="362af1cd20c403341dc49aea3019e9124ac70281f493ae9168393266739f782f" Aug 01 01:26:18.378488 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.378467 2563 generic.go:356] "Generic (PLEG): container finished" podID="382130f6-42bd-45d5-985a-59b9d0b675ff" containerID="3a29b0d8f966fbdb428dd8020b2099d8f115754575ce4192a00154e3793555e1" exitCode=0 Aug 01 01:26:18.378618 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.378516 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" Aug 01 01:26:18.378618 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.378560 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" event={"ID":"382130f6-42bd-45d5-985a-59b9d0b675ff","Type":"ContainerDied","Data":"3a29b0d8f966fbdb428dd8020b2099d8f115754575ce4192a00154e3793555e1"} Aug 01 01:26:18.378618 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.378610 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c" event={"ID":"382130f6-42bd-45d5-985a-59b9d0b675ff","Type":"ContainerDied","Data":"edb5b634fa401e4eab802a3522386265cc657732fdebc8161e967990681a6538"} Aug 01 01:26:18.386833 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.386814 2563 scope.go:117] "RemoveContainer" containerID="362af1cd20c403341dc49aea3019e9124ac70281f493ae9168393266739f782f" Aug 01 01:26:18.387103 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:26:18.387080 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362af1cd20c403341dc49aea3019e9124ac70281f493ae9168393266739f782f\": container with ID starting with 362af1cd20c403341dc49aea3019e9124ac70281f493ae9168393266739f782f not found: ID does not exist" containerID="362af1cd20c403341dc49aea3019e9124ac70281f493ae9168393266739f782f" Aug 01 01:26:18.387228 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.387111 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362af1cd20c403341dc49aea3019e9124ac70281f493ae9168393266739f782f"} err="failed to get container status \"362af1cd20c403341dc49aea3019e9124ac70281f493ae9168393266739f782f\": rpc error: code = NotFound desc = could not find container \"362af1cd20c403341dc49aea3019e9124ac70281f493ae9168393266739f782f\": container with ID starting with 362af1cd20c403341dc49aea3019e9124ac70281f493ae9168393266739f782f not found: ID does not exist" Aug 01 01:26:18.387228 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.387135 2563 scope.go:117] "RemoveContainer" containerID="3a29b0d8f966fbdb428dd8020b2099d8f115754575ce4192a00154e3793555e1" Aug 01 01:26:18.396917 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.396896 2563 scope.go:117] "RemoveContainer" containerID="3a29b0d8f966fbdb428dd8020b2099d8f115754575ce4192a00154e3793555e1" Aug 01 01:26:18.397167 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:26:18.397150 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a29b0d8f966fbdb428dd8020b2099d8f115754575ce4192a00154e3793555e1\": container with ID starting with 3a29b0d8f966fbdb428dd8020b2099d8f115754575ce4192a00154e3793555e1 not found: ID does not exist" containerID="3a29b0d8f966fbdb428dd8020b2099d8f115754575ce4192a00154e3793555e1" Aug 01 01:26:18.397224 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.397173 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a29b0d8f966fbdb428dd8020b2099d8f115754575ce4192a00154e3793555e1"} err="failed to get container status \"3a29b0d8f966fbdb428dd8020b2099d8f115754575ce4192a00154e3793555e1\": rpc error: code = NotFound desc = could not find container \"3a29b0d8f966fbdb428dd8020b2099d8f115754575ce4192a00154e3793555e1\": container with ID starting with 3a29b0d8f966fbdb428dd8020b2099d8f115754575ce4192a00154e3793555e1 not found: ID does not exist" Aug 01 01:26:18.402202 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.402183 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c"] Aug 01 01:26:18.404234 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.404216 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-576e07-bdf579d96-jwc9c"] Aug 01 01:26:18.413025 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.413007 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg"] Aug 01 01:26:18.416180 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:18.416162 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-576e07-7f4ff6ddb8-48vwg"] Aug 01 01:26:19.730863 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:19.730826 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b9b0e49-7a01-4c78-b400-24a49158850b" path="/var/lib/kubelet/pods/1b9b0e49-7a01-4c78-b400-24a49158850b/volumes" Aug 01 01:26:19.731456 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:19.731435 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="382130f6-42bd-45d5-985a-59b9d0b675ff" path="/var/lib/kubelet/pods/382130f6-42bd-45d5-985a-59b9d0b675ff/volumes" Aug 01 01:26:34.868442 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:34.868405 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759"] Aug 01 01:26:34.869445 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:34.869413 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" podUID="56d5fa20-5cb0-4272-a589-5d1ee98a476d" containerName="payload-processing" containerID="cri-o://f65a524646a7b3eef6f03b09e272d2b4c5c81c974c9a95b3ad973ead3af7871e" gracePeriod=30 Aug 01 01:26:34.879493 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:34.878978 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw"] Aug 01 01:26:34.879493 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:34.879226 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" podUID="4f4ceb57-ddbe-484e-8812-a9b00ea82690" containerName="payload-pre-processing" containerID="cri-o://c675d3303420d2ca4cd858dd054e7e8425f2a59781c7239a408f2372a34f736a" gracePeriod=30 Aug 01 01:26:35.150605 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.150577 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" Aug 01 01:26:35.174630 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.174605 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" Aug 01 01:26:35.235796 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.235762 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d5fa20-5cb0-4272-a589-5d1ee98a476d-plugins-config-volume\") pod \"56d5fa20-5cb0-4272-a589-5d1ee98a476d\" (UID: \"56d5fa20-5cb0-4272-a589-5d1ee98a476d\") " Aug 01 01:26:35.235796 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.235801 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4ceb57-ddbe-484e-8812-a9b00ea82690-plugins-config-volume\") pod \"4f4ceb57-ddbe-484e-8812-a9b00ea82690\" (UID: \"4f4ceb57-ddbe-484e-8812-a9b00ea82690\") " Aug 01 01:26:35.236024 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.235890 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvdlk\" (UniqueName: \"kubernetes.io/projected/56d5fa20-5cb0-4272-a589-5d1ee98a476d-kube-api-access-tvdlk\") pod \"56d5fa20-5cb0-4272-a589-5d1ee98a476d\" (UID: \"56d5fa20-5cb0-4272-a589-5d1ee98a476d\") " Aug 01 01:26:35.236024 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.235924 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kblt2\" (UniqueName: \"kubernetes.io/projected/4f4ceb57-ddbe-484e-8812-a9b00ea82690-kube-api-access-kblt2\") pod \"4f4ceb57-ddbe-484e-8812-a9b00ea82690\" (UID: \"4f4ceb57-ddbe-484e-8812-a9b00ea82690\") " Aug 01 01:26:35.236261 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.236233 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d5fa20-5cb0-4272-a589-5d1ee98a476d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56d5fa20-5cb0-4272-a589-5d1ee98a476d" (UID: "56d5fa20-5cb0-4272-a589-5d1ee98a476d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:26:35.236303 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.236279 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4ceb57-ddbe-484e-8812-a9b00ea82690-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f4ceb57-ddbe-484e-8812-a9b00ea82690" (UID: "4f4ceb57-ddbe-484e-8812-a9b00ea82690"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:26:35.236443 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.236419 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d5fa20-5cb0-4272-a589-5d1ee98a476d-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:35.236443 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.236442 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4ceb57-ddbe-484e-8812-a9b00ea82690-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:35.238021 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.238000 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4ceb57-ddbe-484e-8812-a9b00ea82690-kube-api-access-kblt2" (OuterVolumeSpecName: "kube-api-access-kblt2") pod "4f4ceb57-ddbe-484e-8812-a9b00ea82690" (UID: "4f4ceb57-ddbe-484e-8812-a9b00ea82690"). InnerVolumeSpecName "kube-api-access-kblt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:26:35.238101 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.238014 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d5fa20-5cb0-4272-a589-5d1ee98a476d-kube-api-access-tvdlk" (OuterVolumeSpecName: "kube-api-access-tvdlk") pod "56d5fa20-5cb0-4272-a589-5d1ee98a476d" (UID: "56d5fa20-5cb0-4272-a589-5d1ee98a476d"). InnerVolumeSpecName "kube-api-access-tvdlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:26:35.337437 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.337398 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kblt2\" (UniqueName: \"kubernetes.io/projected/4f4ceb57-ddbe-484e-8812-a9b00ea82690-kube-api-access-kblt2\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:35.337437 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.337435 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvdlk\" (UniqueName: \"kubernetes.io/projected/56d5fa20-5cb0-4272-a589-5d1ee98a476d-kube-api-access-tvdlk\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:35.473511 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.473419 2563 generic.go:356] "Generic (PLEG): container finished" podID="56d5fa20-5cb0-4272-a589-5d1ee98a476d" containerID="f65a524646a7b3eef6f03b09e272d2b4c5c81c974c9a95b3ad973ead3af7871e" exitCode=0 Aug 01 01:26:35.473511 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.473460 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" Aug 01 01:26:35.473751 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.473512 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" event={"ID":"56d5fa20-5cb0-4272-a589-5d1ee98a476d","Type":"ContainerDied","Data":"f65a524646a7b3eef6f03b09e272d2b4c5c81c974c9a95b3ad973ead3af7871e"} Aug 01 01:26:35.473751 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.473566 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759" event={"ID":"56d5fa20-5cb0-4272-a589-5d1ee98a476d","Type":"ContainerDied","Data":"4bb4972275bcf79de8d7ab890c015b0eed2f97904b037233fe322224b84510ea"} Aug 01 01:26:35.473751 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.473589 2563 scope.go:117] "RemoveContainer" containerID="f65a524646a7b3eef6f03b09e272d2b4c5c81c974c9a95b3ad973ead3af7871e" Aug 01 01:26:35.474796 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.474765 2563 generic.go:356] "Generic (PLEG): container finished" podID="4f4ceb57-ddbe-484e-8812-a9b00ea82690" containerID="c675d3303420d2ca4cd858dd054e7e8425f2a59781c7239a408f2372a34f736a" exitCode=0 Aug 01 01:26:35.474929 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.474798 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" Aug 01 01:26:35.474929 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.474853 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" event={"ID":"4f4ceb57-ddbe-484e-8812-a9b00ea82690","Type":"ContainerDied","Data":"c675d3303420d2ca4cd858dd054e7e8425f2a59781c7239a408f2372a34f736a"} Aug 01 01:26:35.474929 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.474909 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw" event={"ID":"4f4ceb57-ddbe-484e-8812-a9b00ea82690","Type":"ContainerDied","Data":"ca83157e1f865f29e8ef2898504ac19e68fc60ddf3a3f47e5dba87015cd1eb28"} Aug 01 01:26:35.482232 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.482212 2563 scope.go:117] "RemoveContainer" containerID="f65a524646a7b3eef6f03b09e272d2b4c5c81c974c9a95b3ad973ead3af7871e" Aug 01 01:26:35.482485 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:26:35.482458 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65a524646a7b3eef6f03b09e272d2b4c5c81c974c9a95b3ad973ead3af7871e\": container with ID starting with f65a524646a7b3eef6f03b09e272d2b4c5c81c974c9a95b3ad973ead3af7871e not found: ID does not exist" containerID="f65a524646a7b3eef6f03b09e272d2b4c5c81c974c9a95b3ad973ead3af7871e" Aug 01 01:26:35.482590 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.482493 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65a524646a7b3eef6f03b09e272d2b4c5c81c974c9a95b3ad973ead3af7871e"} err="failed to get container status \"f65a524646a7b3eef6f03b09e272d2b4c5c81c974c9a95b3ad973ead3af7871e\": rpc error: code = NotFound desc = could not find container \"f65a524646a7b3eef6f03b09e272d2b4c5c81c974c9a95b3ad973ead3af7871e\": container with ID starting with f65a524646a7b3eef6f03b09e272d2b4c5c81c974c9a95b3ad973ead3af7871e not found: ID does not exist" Aug 01 01:26:35.482590 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.482516 2563 scope.go:117] "RemoveContainer" containerID="c675d3303420d2ca4cd858dd054e7e8425f2a59781c7239a408f2372a34f736a" Aug 01 01:26:35.490690 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.490673 2563 scope.go:117] "RemoveContainer" containerID="c675d3303420d2ca4cd858dd054e7e8425f2a59781c7239a408f2372a34f736a" Aug 01 01:26:35.490947 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:26:35.490924 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c675d3303420d2ca4cd858dd054e7e8425f2a59781c7239a408f2372a34f736a\": container with ID starting with c675d3303420d2ca4cd858dd054e7e8425f2a59781c7239a408f2372a34f736a not found: ID does not exist" containerID="c675d3303420d2ca4cd858dd054e7e8425f2a59781c7239a408f2372a34f736a" Aug 01 01:26:35.490997 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.490958 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c675d3303420d2ca4cd858dd054e7e8425f2a59781c7239a408f2372a34f736a"} err="failed to get container status \"c675d3303420d2ca4cd858dd054e7e8425f2a59781c7239a408f2372a34f736a\": rpc error: code = NotFound desc = could not find container \"c675d3303420d2ca4cd858dd054e7e8425f2a59781c7239a408f2372a34f736a\": container with ID starting with c675d3303420d2ca4cd858dd054e7e8425f2a59781c7239a408f2372a34f736a not found: ID does not exist" Aug 01 01:26:35.497523 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.497501 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw"] Aug 01 01:26:35.504954 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.504931 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1e2f5b-576b8bcb48-kjhgw"] Aug 01 01:26:35.515029 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.515000 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759"] Aug 01 01:26:35.521146 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.521123 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1e2f5b-78d8bc7666-5r759"] Aug 01 01:26:35.728345 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.728264 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f4ceb57-ddbe-484e-8812-a9b00ea82690" path="/var/lib/kubelet/pods/4f4ceb57-ddbe-484e-8812-a9b00ea82690/volumes" Aug 01 01:26:35.728645 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:35.728632 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56d5fa20-5cb0-4272-a589-5d1ee98a476d" path="/var/lib/kubelet/pods/56d5fa20-5cb0-4272-a589-5d1ee98a476d/volumes" Aug 01 01:26:41.596839 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.596801 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8"] Aug 01 01:26:41.597345 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.597214 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" podUID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" containerName="istio-proxy" containerID="cri-o://46a5d98156a1c1d8449419afecfffcdae0072b2e9305fa11f9eb83d0de02fd3a" gracePeriod=30 Aug 01 01:26:41.979957 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.979878 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs"] Aug 01 01:26:41.980318 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980304 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerName="istio-proxy" Aug 01 01:26:41.980365 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980320 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerName="istio-proxy" Aug 01 01:26:41.980365 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980340 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="382130f6-42bd-45d5-985a-59b9d0b675ff" containerName="payload-processing" Aug 01 01:26:41.980365 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980346 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="382130f6-42bd-45d5-985a-59b9d0b675ff" containerName="payload-processing" Aug 01 01:26:41.980365 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980354 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d5fa20-5cb0-4272-a589-5d1ee98a476d" containerName="payload-processing" Aug 01 01:26:41.980365 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980359 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d5fa20-5cb0-4272-a589-5d1ee98a476d" containerName="payload-processing" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980371 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e51ab2d9-f4ae-47c1-81de-236e5ec3987d" containerName="payload-processing" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980378 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e51ab2d9-f4ae-47c1-81de-236e5ec3987d" containerName="payload-processing" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980388 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b9b0e49-7a01-4c78-b400-24a49158850b" containerName="payload-pre-processing" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980392 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b9b0e49-7a01-4c78-b400-24a49158850b" containerName="payload-pre-processing" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980404 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e690ac14-05c4-4818-a694-3b395d0cf84c" containerName="payload-pre-processing" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980412 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e690ac14-05c4-4818-a694-3b395d0cf84c" containerName="payload-pre-processing" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980420 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f4ceb57-ddbe-484e-8812-a9b00ea82690" containerName="payload-pre-processing" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980424 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f4ceb57-ddbe-484e-8812-a9b00ea82690" containerName="payload-pre-processing" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980484 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerName="istio-proxy" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980496 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d5fa20-5cb0-4272-a589-5d1ee98a476d" containerName="payload-processing" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980503 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e690ac14-05c4-4818-a694-3b395d0cf84c" containerName="payload-pre-processing" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980511 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b9b0e49-7a01-4c78-b400-24a49158850b" containerName="payload-pre-processing" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980515 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e51ab2d9-f4ae-47c1-81de-236e5ec3987d" containerName="payload-processing" Aug 01 01:26:41.980520 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980522 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f4ceb57-ddbe-484e-8812-a9b00ea82690" containerName="payload-pre-processing" Aug 01 01:26:41.980916 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.980528 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="382130f6-42bd-45d5-985a-59b9d0b675ff" containerName="payload-processing" Aug 01 01:26:41.984917 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.984901 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs" Aug 01 01:26:41.988286 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.988261 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68mnr\"" Aug 01 01:26:41.988467 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.988292 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:26:41.990176 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:41.990150 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs"] Aug 01 01:26:42.105191 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:42.105155 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece61706-a832-483a-86bb-db993412e835-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs\" (UID: \"ece61706-a832-483a-86bb-db993412e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs" Aug 01 01:26:42.206465 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:42.206428 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece61706-a832-483a-86bb-db993412e835-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs\" (UID: \"ece61706-a832-483a-86bb-db993412e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs" Aug 01 01:26:42.207078 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:42.207043 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece61706-a832-483a-86bb-db993412e835-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs\" (UID: \"ece61706-a832-483a-86bb-db993412e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs" Aug 01 01:26:42.297713 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:42.297634 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs" Aug 01 01:26:42.428854 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:42.428818 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs"] Aug 01 01:26:42.430788 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:26:42.430760 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece61706_a832_483a_86bb_db993412e835.slice/crio-9b9218de83d1b325b536668e44748707e38f1737c13cdc959e774ff0b0bd2828 WatchSource:0}: Error finding container 9b9218de83d1b325b536668e44748707e38f1737c13cdc959e774ff0b0bd2828: Status 404 returned error can't find the container with id 9b9218de83d1b325b536668e44748707e38f1737c13cdc959e774ff0b0bd2828 Aug 01 01:26:42.515227 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:42.515185 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs" event={"ID":"ece61706-a832-483a-86bb-db993412e835","Type":"ContainerStarted","Data":"9b9218de83d1b325b536668e44748707e38f1737c13cdc959e774ff0b0bd2828"} Aug 01 01:26:43.208724 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:43.208684 2563 patch_prober.go:28] interesting pod/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 01 01:26:43.209319 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:43.208757 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" podUID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 01 01:26:43.523153 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:43.523041 2563 generic.go:356] "Generic (PLEG): container finished" podID="ece61706-a832-483a-86bb-db993412e835" containerID="a40b053355e355d553a36b821700906bf14cb8dbbc9077741d3873b009d3cb4c" exitCode=0 Aug 01 01:26:43.523153 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:43.523125 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs" event={"ID":"ece61706-a832-483a-86bb-db993412e835","Type":"ContainerDied","Data":"a40b053355e355d553a36b821700906bf14cb8dbbc9077741d3873b009d3cb4c"} Aug 01 01:26:44.660724 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:44.660700 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs" Aug 01 01:26:44.732903 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:44.732867 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece61706-a832-483a-86bb-db993412e835-maas-api-service-ca\") pod \"ece61706-a832-483a-86bb-db993412e835\" (UID: \"ece61706-a832-483a-86bb-db993412e835\") " Aug 01 01:26:44.733280 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:44.733257 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece61706-a832-483a-86bb-db993412e835-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ece61706-a832-483a-86bb-db993412e835" (UID: "ece61706-a832-483a-86bb-db993412e835"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:26:44.833893 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:44.833860 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece61706-a832-483a-86bb-db993412e835-maas-api-service-ca\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:45.534657 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:45.534627 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs" Aug 01 01:26:45.534825 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:45.534627 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs" event={"ID":"ece61706-a832-483a-86bb-db993412e835","Type":"ContainerDied","Data":"9b9218de83d1b325b536668e44748707e38f1737c13cdc959e774ff0b0bd2828"} Aug 01 01:26:45.534825 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:45.534782 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b9218de83d1b325b536668e44748707e38f1737c13cdc959e774ff0b0bd2828" Aug 01 01:26:46.837487 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.837460 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:26:46.955373 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.955291 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istiod-ca-cert\") pod \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " Aug 01 01:26:46.955373 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.955344 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-token\") pod \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " Aug 01 01:26:46.955600 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.955377 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-credential-socket\") pod \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " Aug 01 01:26:46.955600 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.955416 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-data\") pod \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " Aug 01 01:26:46.955600 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.955459 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwpfw\" (UniqueName: \"kubernetes.io/projected/9558c6fe-3d0a-4915-ae05-7d6701b3891d-kube-api-access-nwpfw\") pod \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " Aug 01 01:26:46.955600 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.955551 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-envoy\") pod \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " Aug 01 01:26:46.955600 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.955584 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-workload-socket\") pod \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " Aug 01 01:26:46.955846 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.955617 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-workload-certs\") pod \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " Aug 01 01:26:46.955846 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.955662 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-podinfo\") pod \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\" (UID: \"9558c6fe-3d0a-4915-ae05-7d6701b3891d\") " Aug 01 01:26:46.955846 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.955723 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-data" (OuterVolumeSpecName: "istio-data") pod "9558c6fe-3d0a-4915-ae05-7d6701b3891d" (UID: "9558c6fe-3d0a-4915-ae05-7d6701b3891d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:26:46.955846 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.955765 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9558c6fe-3d0a-4915-ae05-7d6701b3891d" (UID: "9558c6fe-3d0a-4915-ae05-7d6701b3891d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:26:46.956042 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.955875 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9558c6fe-3d0a-4915-ae05-7d6701b3891d" (UID: "9558c6fe-3d0a-4915-ae05-7d6701b3891d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:26:46.956042 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.955957 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9558c6fe-3d0a-4915-ae05-7d6701b3891d" (UID: "9558c6fe-3d0a-4915-ae05-7d6701b3891d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:26:46.956179 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.956038 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-workload-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:46.956179 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.956026 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9558c6fe-3d0a-4915-ae05-7d6701b3891d" (UID: "9558c6fe-3d0a-4915-ae05-7d6701b3891d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:26:46.956179 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.956084 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-workload-certs\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:46.956179 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.956099 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istiod-ca-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:46.956179 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.956108 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-data\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:46.957769 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.957738 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9558c6fe-3d0a-4915-ae05-7d6701b3891d" (UID: "9558c6fe-3d0a-4915-ae05-7d6701b3891d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:26:46.957877 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.957793 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9558c6fe-3d0a-4915-ae05-7d6701b3891d-kube-api-access-nwpfw" (OuterVolumeSpecName: "kube-api-access-nwpfw") pod "9558c6fe-3d0a-4915-ae05-7d6701b3891d" (UID: "9558c6fe-3d0a-4915-ae05-7d6701b3891d"). InnerVolumeSpecName "kube-api-access-nwpfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:26:46.958023 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.957994 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9558c6fe-3d0a-4915-ae05-7d6701b3891d" (UID: "9558c6fe-3d0a-4915-ae05-7d6701b3891d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 01 01:26:46.958023 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:46.957999 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-token" (OuterVolumeSpecName: "istio-token") pod "9558c6fe-3d0a-4915-ae05-7d6701b3891d" (UID: "9558c6fe-3d0a-4915-ae05-7d6701b3891d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:26:47.056770 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.056739 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-envoy\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:47.056770 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.056765 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-podinfo\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:47.056770 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.056774 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9558c6fe-3d0a-4915-ae05-7d6701b3891d-istio-token\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:47.056973 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.056783 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9558c6fe-3d0a-4915-ae05-7d6701b3891d-credential-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:47.056973 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.056792 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwpfw\" (UniqueName: \"kubernetes.io/projected/9558c6fe-3d0a-4915-ae05-7d6701b3891d-kube-api-access-nwpfw\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:47.545732 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.545694 2563 generic.go:356] "Generic (PLEG): container finished" podID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" containerID="46a5d98156a1c1d8449419afecfffcdae0072b2e9305fa11f9eb83d0de02fd3a" exitCode=0 Aug 01 01:26:47.545903 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.545738 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" Aug 01 01:26:47.545903 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.545769 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" event={"ID":"9558c6fe-3d0a-4915-ae05-7d6701b3891d","Type":"ContainerDied","Data":"46a5d98156a1c1d8449419afecfffcdae0072b2e9305fa11f9eb83d0de02fd3a"} Aug 01 01:26:47.545903 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.545816 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8" event={"ID":"9558c6fe-3d0a-4915-ae05-7d6701b3891d","Type":"ContainerDied","Data":"718da4a59d316b37ed5986ceace9071f1bedd0d72ec32aa4a7af0851b019b62e"} Aug 01 01:26:47.545903 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.545840 2563 scope.go:117] "RemoveContainer" containerID="46a5d98156a1c1d8449419afecfffcdae0072b2e9305fa11f9eb83d0de02fd3a" Aug 01 01:26:47.555139 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.555123 2563 scope.go:117] "RemoveContainer" containerID="9d7c8e097422b1abfbaa81f13de48a160b7414dd1c757d37181210aa52da6eba" Aug 01 01:26:47.562939 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.562920 2563 scope.go:117] "RemoveContainer" containerID="46a5d98156a1c1d8449419afecfffcdae0072b2e9305fa11f9eb83d0de02fd3a" Aug 01 01:26:47.563270 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:26:47.563251 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a5d98156a1c1d8449419afecfffcdae0072b2e9305fa11f9eb83d0de02fd3a\": container with ID starting with 46a5d98156a1c1d8449419afecfffcdae0072b2e9305fa11f9eb83d0de02fd3a not found: ID does not exist" containerID="46a5d98156a1c1d8449419afecfffcdae0072b2e9305fa11f9eb83d0de02fd3a" Aug 01 01:26:47.563325 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.563277 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a5d98156a1c1d8449419afecfffcdae0072b2e9305fa11f9eb83d0de02fd3a"} err="failed to get container status \"46a5d98156a1c1d8449419afecfffcdae0072b2e9305fa11f9eb83d0de02fd3a\": rpc error: code = NotFound desc = could not find container \"46a5d98156a1c1d8449419afecfffcdae0072b2e9305fa11f9eb83d0de02fd3a\": container with ID starting with 46a5d98156a1c1d8449419afecfffcdae0072b2e9305fa11f9eb83d0de02fd3a not found: ID does not exist" Aug 01 01:26:47.563325 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.563293 2563 scope.go:117] "RemoveContainer" containerID="9d7c8e097422b1abfbaa81f13de48a160b7414dd1c757d37181210aa52da6eba" Aug 01 01:26:47.563523 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:26:47.563505 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7c8e097422b1abfbaa81f13de48a160b7414dd1c757d37181210aa52da6eba\": container with ID starting with 9d7c8e097422b1abfbaa81f13de48a160b7414dd1c757d37181210aa52da6eba not found: ID does not exist" containerID="9d7c8e097422b1abfbaa81f13de48a160b7414dd1c757d37181210aa52da6eba" Aug 01 01:26:47.563575 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.563532 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7c8e097422b1abfbaa81f13de48a160b7414dd1c757d37181210aa52da6eba"} err="failed to get container status \"9d7c8e097422b1abfbaa81f13de48a160b7414dd1c757d37181210aa52da6eba\": rpc error: code = NotFound desc = could not find container \"9d7c8e097422b1abfbaa81f13de48a160b7414dd1c757d37181210aa52da6eba\": container with ID starting with 9d7c8e097422b1abfbaa81f13de48a160b7414dd1c757d37181210aa52da6eba not found: ID does not exist" Aug 01 01:26:47.566677 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.566657 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8"] Aug 01 01:26:47.570929 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.570908 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-1e2f5b-openshift-default-5474bd75cb-44fz8"] Aug 01 01:26:47.728298 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:47.728263 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" path="/var/lib/kubelet/pods/9558c6fe-3d0a-4915-ae05-7d6701b3891d/volumes" Aug 01 01:26:52.580260 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:52.580218 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r"] Aug 01 01:26:52.580696 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:52.580473 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" podUID="71ab65c0-6274-4804-b03f-7cf02765bddb" containerName="payload-processing" containerID="cri-o://9a8b465ea3694be4867acf34d7770c088a2e9ae00712feb70ee170ab9d527c0e" gracePeriod=30 Aug 01 01:26:52.716554 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:52.716496 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs"] Aug 01 01:26:52.718488 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:52.718459 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4224e1-qjghs"] Aug 01 01:26:52.840092 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:52.840027 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" Aug 01 01:26:53.019268 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.019218 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ab65c0-6274-4804-b03f-7cf02765bddb-plugins-config-volume\") pod \"71ab65c0-6274-4804-b03f-7cf02765bddb\" (UID: \"71ab65c0-6274-4804-b03f-7cf02765bddb\") " Aug 01 01:26:53.019473 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.019312 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwgwh\" (UniqueName: \"kubernetes.io/projected/71ab65c0-6274-4804-b03f-7cf02765bddb-kube-api-access-xwgwh\") pod \"71ab65c0-6274-4804-b03f-7cf02765bddb\" (UID: \"71ab65c0-6274-4804-b03f-7cf02765bddb\") " Aug 01 01:26:53.019684 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.019662 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ab65c0-6274-4804-b03f-7cf02765bddb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71ab65c0-6274-4804-b03f-7cf02765bddb" (UID: "71ab65c0-6274-4804-b03f-7cf02765bddb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:26:53.021432 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.021405 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ab65c0-6274-4804-b03f-7cf02765bddb-kube-api-access-xwgwh" (OuterVolumeSpecName: "kube-api-access-xwgwh") pod "71ab65c0-6274-4804-b03f-7cf02765bddb" (UID: "71ab65c0-6274-4804-b03f-7cf02765bddb"). InnerVolumeSpecName "kube-api-access-xwgwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:26:53.121111 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.120992 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ab65c0-6274-4804-b03f-7cf02765bddb-plugins-config-volume\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:53.121111 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.121016 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwgwh\" (UniqueName: \"kubernetes.io/projected/71ab65c0-6274-4804-b03f-7cf02765bddb-kube-api-access-xwgwh\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:26:53.577045 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.577012 2563 generic.go:356] "Generic (PLEG): container finished" podID="71ab65c0-6274-4804-b03f-7cf02765bddb" containerID="9a8b465ea3694be4867acf34d7770c088a2e9ae00712feb70ee170ab9d527c0e" exitCode=0 Aug 01 01:26:53.577225 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.577053 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" Aug 01 01:26:53.577225 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.577091 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" event={"ID":"71ab65c0-6274-4804-b03f-7cf02765bddb","Type":"ContainerDied","Data":"9a8b465ea3694be4867acf34d7770c088a2e9ae00712feb70ee170ab9d527c0e"} Aug 01 01:26:53.577225 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.577144 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r" event={"ID":"71ab65c0-6274-4804-b03f-7cf02765bddb","Type":"ContainerDied","Data":"30d414f4a17919017f782672cc26f9ac7b68545dc320ea097b677649f84e867e"} Aug 01 01:26:53.577225 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.577167 2563 scope.go:117] "RemoveContainer" containerID="9a8b465ea3694be4867acf34d7770c088a2e9ae00712feb70ee170ab9d527c0e" Aug 01 01:26:53.585703 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.585482 2563 scope.go:117] "RemoveContainer" containerID="9a8b465ea3694be4867acf34d7770c088a2e9ae00712feb70ee170ab9d527c0e" Aug 01 01:26:53.585929 ip-10-0-128-195 kubenswrapper[2563]: E0801 01:26:53.585762 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a8b465ea3694be4867acf34d7770c088a2e9ae00712feb70ee170ab9d527c0e\": container with ID starting with 9a8b465ea3694be4867acf34d7770c088a2e9ae00712feb70ee170ab9d527c0e not found: ID does not exist" containerID="9a8b465ea3694be4867acf34d7770c088a2e9ae00712feb70ee170ab9d527c0e" Aug 01 01:26:53.585929 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.585796 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8b465ea3694be4867acf34d7770c088a2e9ae00712feb70ee170ab9d527c0e"} err="failed to get container status \"9a8b465ea3694be4867acf34d7770c088a2e9ae00712feb70ee170ab9d527c0e\": rpc error: code = NotFound desc = could not find container \"9a8b465ea3694be4867acf34d7770c088a2e9ae00712feb70ee170ab9d527c0e\": container with ID starting with 9a8b465ea3694be4867acf34d7770c088a2e9ae00712feb70ee170ab9d527c0e not found: ID does not exist" Aug 01 01:26:53.600525 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.600496 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r"] Aug 01 01:26:53.605089 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.605051 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4224e1-74d6cb8798-jtd9r"] Aug 01 01:26:53.729603 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.729574 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ab65c0-6274-4804-b03f-7cf02765bddb" path="/var/lib/kubelet/pods/71ab65c0-6274-4804-b03f-7cf02765bddb/volumes" Aug 01 01:26:53.729885 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:53.729873 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ece61706-a832-483a-86bb-db993412e835" path="/var/lib/kubelet/pods/ece61706-a832-483a-86bb-db993412e835/volumes" Aug 01 01:26:59.422178 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:59.422141 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk"] Aug 01 01:26:59.422745 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:26:59.422402 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" podUID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerName="istio-proxy" containerID="cri-o://30e6ad3cf141b5d79d325c347c92d6588c5b420a0d68a3582860fee562ffb7c2" gracePeriod=30 Aug 01 01:27:04.633349 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.633315 2563 generic.go:356] "Generic (PLEG): container finished" podID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerID="30e6ad3cf141b5d79d325c347c92d6588c5b420a0d68a3582860fee562ffb7c2" exitCode=0 Aug 01 01:27:04.633702 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.633391 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" event={"ID":"042f0d0f-3699-44ae-bbfb-e4f842739398","Type":"ContainerDied","Data":"30e6ad3cf141b5d79d325c347c92d6588c5b420a0d68a3582860fee562ffb7c2"} Aug 01 01:27:04.633702 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.633458 2563 scope.go:117] "RemoveContainer" containerID="436fb01adaf18253b0e3b1f21dbb26ecb1f280a9997c426eed698e1f5c226501" Aug 01 01:27:04.673292 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.673271 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:27:04.724074 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.723995 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-workload-certs\") pod \"042f0d0f-3699-44ae-bbfb-e4f842739398\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " Aug 01 01:27:04.724074 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724040 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-data\") pod \"042f0d0f-3699-44ae-bbfb-e4f842739398\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " Aug 01 01:27:04.724259 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724100 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-podinfo\") pod \"042f0d0f-3699-44ae-bbfb-e4f842739398\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " Aug 01 01:27:04.724259 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724139 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-workload-socket\") pod \"042f0d0f-3699-44ae-bbfb-e4f842739398\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " Aug 01 01:27:04.724259 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724190 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-credential-socket\") pod \"042f0d0f-3699-44ae-bbfb-e4f842739398\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " Aug 01 01:27:04.724259 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724227 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/042f0d0f-3699-44ae-bbfb-e4f842739398-istiod-ca-cert\") pod \"042f0d0f-3699-44ae-bbfb-e4f842739398\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " Aug 01 01:27:04.724259 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724255 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbdd2\" (UniqueName: \"kubernetes.io/projected/042f0d0f-3699-44ae-bbfb-e4f842739398-kube-api-access-cbdd2\") pod \"042f0d0f-3699-44ae-bbfb-e4f842739398\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " Aug 01 01:27:04.724567 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724279 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-envoy\") pod \"042f0d0f-3699-44ae-bbfb-e4f842739398\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " Aug 01 01:27:04.724567 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724286 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "042f0d0f-3699-44ae-bbfb-e4f842739398" (UID: "042f0d0f-3699-44ae-bbfb-e4f842739398"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:27:04.724567 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724307 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-token\") pod \"042f0d0f-3699-44ae-bbfb-e4f842739398\" (UID: \"042f0d0f-3699-44ae-bbfb-e4f842739398\") " Aug 01 01:27:04.724567 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724324 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-data" (OuterVolumeSpecName: "istio-data") pod "042f0d0f-3699-44ae-bbfb-e4f842739398" (UID: "042f0d0f-3699-44ae-bbfb-e4f842739398"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:27:04.724567 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724433 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "042f0d0f-3699-44ae-bbfb-e4f842739398" (UID: "042f0d0f-3699-44ae-bbfb-e4f842739398"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:27:04.724567 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724479 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "042f0d0f-3699-44ae-bbfb-e4f842739398" (UID: "042f0d0f-3699-44ae-bbfb-e4f842739398"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:27:04.724878 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724582 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-workload-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:27:04.724878 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724595 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-credential-socket\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:27:04.724878 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724606 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-workload-certs\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:27:04.724878 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724614 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-data\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:27:04.724878 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.724612 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/042f0d0f-3699-44ae-bbfb-e4f842739398-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "042f0d0f-3699-44ae-bbfb-e4f842739398" (UID: "042f0d0f-3699-44ae-bbfb-e4f842739398"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:27:04.726388 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.726364 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-token" (OuterVolumeSpecName: "istio-token") pod "042f0d0f-3699-44ae-bbfb-e4f842739398" (UID: "042f0d0f-3699-44ae-bbfb-e4f842739398"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:27:04.726388 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.726379 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "042f0d0f-3699-44ae-bbfb-e4f842739398" (UID: "042f0d0f-3699-44ae-bbfb-e4f842739398"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:27:04.726519 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.726441 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "042f0d0f-3699-44ae-bbfb-e4f842739398" (UID: "042f0d0f-3699-44ae-bbfb-e4f842739398"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 01 01:27:04.726682 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.726667 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042f0d0f-3699-44ae-bbfb-e4f842739398-kube-api-access-cbdd2" (OuterVolumeSpecName: "kube-api-access-cbdd2") pod "042f0d0f-3699-44ae-bbfb-e4f842739398" (UID: "042f0d0f-3699-44ae-bbfb-e4f842739398"). InnerVolumeSpecName "kube-api-access-cbdd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:27:04.825386 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.825362 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-podinfo\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:27:04.825386 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.825382 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/042f0d0f-3699-44ae-bbfb-e4f842739398-istiod-ca-cert\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:27:04.825553 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.825392 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbdd2\" (UniqueName: \"kubernetes.io/projected/042f0d0f-3699-44ae-bbfb-e4f842739398-kube-api-access-cbdd2\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:27:04.825553 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.825401 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-envoy\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:27:04.825553 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:04.825409 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/042f0d0f-3699-44ae-bbfb-e4f842739398-istio-token\") on node \"ip-10-0-128-195.ec2.internal\" DevicePath \"\"" Aug 01 01:27:05.638866 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:05.638832 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" event={"ID":"042f0d0f-3699-44ae-bbfb-e4f842739398","Type":"ContainerDied","Data":"2395326c734cae09001553892298fe99061824cb3f88cf7ee0bbc947b274f4c4"} Aug 01 01:27:05.638866 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:05.638860 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk" Aug 01 01:27:05.639440 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:05.638887 2563 scope.go:117] "RemoveContainer" containerID="30e6ad3cf141b5d79d325c347c92d6588c5b420a0d68a3582860fee562ffb7c2" Aug 01 01:27:05.661281 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:05.661257 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk"] Aug 01 01:27:05.664504 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:05.664483 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-4224e1-openshift-default-66bc8c5644-57thk"] Aug 01 01:27:05.727749 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:05.727722 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="042f0d0f-3699-44ae-bbfb-e4f842739398" path="/var/lib/kubelet/pods/042f0d0f-3699-44ae-bbfb-e4f842739398/volumes" Aug 01 01:27:05.772974 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:05.772951 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f44b49488-lbqlf_80beabaa-abdf-414b-8869-f6e808e11faa/manager/0.log" Aug 01 01:27:06.243034 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:06.243008 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c56689d88-dgxwp_27a0d7fa-1f37-4099-8e34-cede1ae3dba2/copy-manifests/0.log" Aug 01 01:27:06.254365 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:06.254336 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c56689d88-dgxwp_27a0d7fa-1f37-4099-8e34-cede1ae3dba2/manager/0.log" Aug 01 01:27:06.830592 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:06.830567 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29759085-bvnz8_c787ccd8-b889-48ec-afe2-3c900e24c4e9/cleanup/0.log" Aug 01 01:27:08.447324 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:08.447293 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-cz2lj_c51dd1bf-0657-4eb0-a8ee-53581202dc5f/manager/0.log" Aug 01 01:27:09.126786 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:09.126748 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rc6mv_3c28f4c7-2878-4ef2-819c-c72cf78034d1/discovery/0.log" Aug 01 01:27:09.578718 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:09.578690 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-9bm8s_0568d048-0ee5-4e6d-a26e-416732ccdc02/payload-pre-processing/0.log" Aug 01 01:27:10.182806 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:10.182779 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-wlzb4_37c3dc90-dd65-45f4-9ff4-ed81c7baeba1/main/0.log" Aug 01 01:27:10.655514 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:10.655485 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-4l7vz_fdda65f4-0d04-44ec-9f42-5dc43126a777/main/0.log" Aug 01 01:27:18.829930 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:18.829900 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nlw9l_df497cda-451a-4e65-b0a8-b18769d3c9d2/global-pull-secret-syncer/0.log" Aug 01 01:27:18.920618 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:18.920594 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sbb29_251336a2-faf4-46f3-b301-ea716514b009/konnectivity-agent/0.log" Aug 01 01:27:18.985202 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:18.985175 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-195.ec2.internal_bc8c8050d8f35c84ce46fb25d5f43d30/haproxy/0.log" Aug 01 01:27:24.205724 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:24.205693 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-cz2lj_c51dd1bf-0657-4eb0-a8ee-53581202dc5f/manager/0.log" Aug 01 01:27:25.973462 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:25.973419 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/1.log" Aug 01 01:27:26.043489 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.043462 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-znmsh_802cbf7b-10e0-4ff6-8443-c0f7b60a9727/cluster-monitoring-operator/0.log" Aug 01 01:27:26.091772 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.091687 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-h9tms_6b43079d-4e13-4651-98a3-47a97f87fd82/kube-state-metrics/0.log" Aug 01 01:27:26.118141 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.118115 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-h9tms_6b43079d-4e13-4651-98a3-47a97f87fd82/kube-rbac-proxy-main/0.log" Aug 01 01:27:26.145763 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.145742 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-h9tms_6b43079d-4e13-4651-98a3-47a97f87fd82/kube-rbac-proxy-self/0.log" Aug 01 01:27:26.178778 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.178736 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-65d6c48b96-kr2xn_01a2397b-9138-42f4-b397-4eb2705829d3/metrics-server/0.log" Aug 01 01:27:26.212762 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.212743 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-vsshj_dd7c88f4-a5b2-45bd-9a87-78a902daae7c/monitoring-plugin/0.log" Aug 01 01:27:26.402629 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.402557 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mvkhk_33ed01ee-85e2-444b-b942-93f4c2393b35/node-exporter/0.log" Aug 01 01:27:26.422593 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.422571 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mvkhk_33ed01ee-85e2-444b-b942-93f4c2393b35/kube-rbac-proxy/0.log" Aug 01 01:27:26.443544 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.443520 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mvkhk_33ed01ee-85e2-444b-b942-93f4c2393b35/init-textfile/0.log" Aug 01 01:27:26.584207 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.584178 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48998805-0245-4641-a435-23d88664daca/prometheus/0.log" Aug 01 01:27:26.606034 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.606013 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48998805-0245-4641-a435-23d88664daca/config-reloader/0.log" Aug 01 01:27:26.627435 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.627378 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48998805-0245-4641-a435-23d88664daca/thanos-sidecar/0.log" Aug 01 01:27:26.645257 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.645233 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48998805-0245-4641-a435-23d88664daca/kube-rbac-proxy-web/0.log" Aug 01 01:27:26.665069 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.664994 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48998805-0245-4641-a435-23d88664daca/kube-rbac-proxy/0.log" Aug 01 01:27:26.683590 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.683568 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48998805-0245-4641-a435-23d88664daca/kube-rbac-proxy-thanos/0.log" Aug 01 01:27:26.701817 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.701797 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48998805-0245-4641-a435-23d88664daca/init-config-reloader/0.log" Aug 01 01:27:26.877067 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.877022 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75755945d6-p6b8l_3f3349b2-3e23-48e3-89d8-76e3fdb6d354/thanos-query/0.log" Aug 01 01:27:26.898293 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.898273 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75755945d6-p6b8l_3f3349b2-3e23-48e3-89d8-76e3fdb6d354/kube-rbac-proxy-web/0.log" Aug 01 01:27:26.919781 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.919723 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75755945d6-p6b8l_3f3349b2-3e23-48e3-89d8-76e3fdb6d354/kube-rbac-proxy/0.log" Aug 01 01:27:26.940624 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.940599 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75755945d6-p6b8l_3f3349b2-3e23-48e3-89d8-76e3fdb6d354/prom-label-proxy/0.log" Aug 01 01:27:26.959345 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.959324 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75755945d6-p6b8l_3f3349b2-3e23-48e3-89d8-76e3fdb6d354/kube-rbac-proxy-rules/0.log" Aug 01 01:27:26.978723 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:26.978702 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75755945d6-p6b8l_3f3349b2-3e23-48e3-89d8-76e3fdb6d354/kube-rbac-proxy-metrics/0.log" Aug 01 01:27:29.115277 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115242 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7"] Aug 01 01:27:29.115663 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115635 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece61706-a832-483a-86bb-db993412e835" containerName="revoke-keys" Aug 01 01:27:29.115663 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115647 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece61706-a832-483a-86bb-db993412e835" containerName="revoke-keys" Aug 01 01:27:29.115663 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115663 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" containerName="istio-proxy" Aug 01 01:27:29.115772 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115668 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" containerName="istio-proxy" Aug 01 01:27:29.115772 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115682 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerName="istio-proxy" Aug 01 01:27:29.115772 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115688 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerName="istio-proxy" Aug 01 01:27:29.115772 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115698 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ab65c0-6274-4804-b03f-7cf02765bddb" containerName="payload-processing" Aug 01 01:27:29.115772 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115703 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ab65c0-6274-4804-b03f-7cf02765bddb" containerName="payload-processing" Aug 01 01:27:29.115772 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115708 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerName="istio-proxy" Aug 01 01:27:29.115772 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115713 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerName="istio-proxy" Aug 01 01:27:29.115772 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115722 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerName="istio-proxy" Aug 01 01:27:29.115772 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115727 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerName="istio-proxy" Aug 01 01:27:29.116034 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115778 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9558c6fe-3d0a-4915-ae05-7d6701b3891d" containerName="istio-proxy" Aug 01 01:27:29.116034 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115785 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de5ae02-8385-40fe-8803-08452b9fd73b" containerName="istio-proxy" Aug 01 01:27:29.116034 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115795 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ab65c0-6274-4804-b03f-7cf02765bddb" containerName="payload-processing" Aug 01 01:27:29.116034 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115800 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerName="istio-proxy" Aug 01 01:27:29.116034 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115806 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="042f0d0f-3699-44ae-bbfb-e4f842739398" containerName="istio-proxy" Aug 01 01:27:29.116034 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.115812 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece61706-a832-483a-86bb-db993412e835" containerName="revoke-keys" Aug 01 01:27:29.118401 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.118384 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.120813 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.120795 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bl84r\"/\"kube-root-ca.crt\"" Aug 01 01:27:29.121738 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.121722 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bl84r\"/\"openshift-service-ca.crt\"" Aug 01 01:27:29.121798 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.121748 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bl84r\"/\"default-dockercfg-hwcw7\"" Aug 01 01:27:29.128583 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.128562 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7"] Aug 01 01:27:29.222668 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.222639 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eeb88483-a7fd-443c-baa0-f307f54fedf4-podres\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.222811 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.222677 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eeb88483-a7fd-443c-baa0-f307f54fedf4-lib-modules\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.222811 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.222766 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eeb88483-a7fd-443c-baa0-f307f54fedf4-proc\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.222811 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.222794 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeb88483-a7fd-443c-baa0-f307f54fedf4-sys\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.222959 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.222818 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p287c\" (UniqueName: \"kubernetes.io/projected/eeb88483-a7fd-443c-baa0-f307f54fedf4-kube-api-access-p287c\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.323500 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.323466 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eeb88483-a7fd-443c-baa0-f307f54fedf4-podres\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.323682 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.323506 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eeb88483-a7fd-443c-baa0-f307f54fedf4-lib-modules\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.323682 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.323558 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eeb88483-a7fd-443c-baa0-f307f54fedf4-proc\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.323682 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.323572 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeb88483-a7fd-443c-baa0-f307f54fedf4-sys\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.323682 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.323593 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p287c\" (UniqueName: \"kubernetes.io/projected/eeb88483-a7fd-443c-baa0-f307f54fedf4-kube-api-access-p287c\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.323682 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.323625 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eeb88483-a7fd-443c-baa0-f307f54fedf4-podres\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.323682 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.323663 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eeb88483-a7fd-443c-baa0-f307f54fedf4-lib-modules\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.323682 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.323674 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeb88483-a7fd-443c-baa0-f307f54fedf4-sys\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.323951 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.323689 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eeb88483-a7fd-443c-baa0-f307f54fedf4-proc\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.330844 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.330819 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p287c\" (UniqueName: \"kubernetes.io/projected/eeb88483-a7fd-443c-baa0-f307f54fedf4-kube-api-access-p287c\") pod \"perf-node-gather-daemonset-xgnk7\" (UID: \"eeb88483-a7fd-443c-baa0-f307f54fedf4\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.429146 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.429045 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.558354 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.558331 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7"] Aug 01 01:27:29.560457 ip-10-0-128-195 kubenswrapper[2563]: W0801 01:27:29.560430 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeeb88483_a7fd_443c_baa0_f307f54fedf4.slice/crio-e33735178660146800975298d34b33d20957011b64545bc7f6205ce0e6a8ea17 WatchSource:0}: Error finding container e33735178660146800975298d34b33d20957011b64545bc7f6205ce0e6a8ea17: Status 404 returned error can't find the container with id e33735178660146800975298d34b33d20957011b64545bc7f6205ce0e6a8ea17 Aug 01 01:27:29.754449 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.754363 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" event={"ID":"eeb88483-a7fd-443c-baa0-f307f54fedf4","Type":"ContainerStarted","Data":"9b1059080cdb1b5dc8a53516e59b41e579ba7a1e50f03a05c4040cedef95d744"} Aug 01 01:27:29.754449 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.754415 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" event={"ID":"eeb88483-a7fd-443c-baa0-f307f54fedf4","Type":"ContainerStarted","Data":"e33735178660146800975298d34b33d20957011b64545bc7f6205ce0e6a8ea17"} Aug 01 01:27:29.754626 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.754498 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:29.771907 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:29.771858 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" podStartSLOduration=0.771844615 podStartE2EDuration="771.844615ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 01:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:27:29.770364927 +0000 UTC m=+3538.559086768" watchObservedRunningTime="2026-08-01 01:27:29.771844615 +0000 UTC m=+3538.560566441" Aug 01 01:27:31.155472 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:31.155445 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7hj6_e5b71018-3dd9-43cc-88e1-220d978adcf6/dns/0.log" Aug 01 01:27:31.175162 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:31.175139 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7hj6_e5b71018-3dd9-43cc-88e1-220d978adcf6/kube-rbac-proxy/0.log" Aug 01 01:27:31.240690 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:31.240671 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5b74f_a434193d-a324-4f56-90ae-82de780986a4/dns-node-resolver/0.log" Aug 01 01:27:31.830492 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:31.830463 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8w9d6_fee8d179-f489-4751-895d-b5294aa866fb/node-ca/0.log" Aug 01 01:27:32.938918 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:32.938882 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rc6mv_3c28f4c7-2878-4ef2-819c-c72cf78034d1/discovery/0.log" Aug 01 01:27:33.047342 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:33.047317 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-9bm8s_0568d048-0ee5-4e6d-a26e-416732ccdc02/payload-pre-processing/0.log" Aug 01 01:27:33.930392 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:33.930361 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5kstw_1a6ef2cf-9383-4287-8b3d-5fc395998c22/serve-healthcheck-canary/0.log" Aug 01 01:27:34.390200 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:34.390173 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-nlksz_a878cee9-835b-4d43-9d83-b5469f709610/insights-operator/0.log" Aug 01 01:27:34.390567 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:34.390288 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-nlksz_a878cee9-835b-4d43-9d83-b5469f709610/insights-operator/1.log" Aug 01 01:27:34.410672 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:34.410651 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8x7hp_e52c1e00-286c-4c75-b6c1-54481a442121/kube-rbac-proxy/0.log" Aug 01 01:27:34.442486 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:34.442463 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8x7hp_e52c1e00-286c-4c75-b6c1-54481a442121/exporter/0.log" Aug 01 01:27:34.462739 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:34.462717 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8x7hp_e52c1e00-286c-4c75-b6c1-54481a442121/extractor/0.log" Aug 01 01:27:35.767938 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:35.767912 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-xgnk7" Aug 01 01:27:37.087380 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:37.087346 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f44b49488-lbqlf_80beabaa-abdf-414b-8869-f6e808e11faa/manager/0.log" Aug 01 01:27:37.326443 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:37.326411 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c56689d88-dgxwp_27a0d7fa-1f37-4099-8e34-cede1ae3dba2/manager/0.log" Aug 01 01:27:37.341859 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:37.341798 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c56689d88-dgxwp_27a0d7fa-1f37-4099-8e34-cede1ae3dba2/copy-manifests/0.log" Aug 01 01:27:38.812502 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:38.812481 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5dd57cb99d-tpqcq_074fa142-2156-4b3b-937b-c9ed2b811e4b/manager/0.log" Aug 01 01:27:39.541760 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:39.541684 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rc6mv_3c28f4c7-2878-4ef2-819c-c72cf78034d1/discovery/0.log" Aug 01 01:27:39.624312 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:39.624283 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-9bm8s_0568d048-0ee5-4e6d-a26e-416732ccdc02/payload-pre-processing/0.log" Aug 01 01:27:41.218919 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:41.218879 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rc6mv_3c28f4c7-2878-4ef2-819c-c72cf78034d1/discovery/0.log" Aug 01 01:27:41.299348 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:41.299322 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-9bm8s_0568d048-0ee5-4e6d-a26e-416732ccdc02/payload-pre-processing/0.log" Aug 01 01:27:45.518747 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:45.518717 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q2wgc_a4e452e0-a47b-4d14-8852-17f8567c98fc/kube-multus-additional-cni-plugins/0.log" Aug 01 01:27:45.537901 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:45.537877 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q2wgc_a4e452e0-a47b-4d14-8852-17f8567c98fc/egress-router-binary-copy/0.log" Aug 01 01:27:45.556715 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:45.556683 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q2wgc_a4e452e0-a47b-4d14-8852-17f8567c98fc/cni-plugins/0.log" Aug 01 01:27:45.576936 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:45.576917 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q2wgc_a4e452e0-a47b-4d14-8852-17f8567c98fc/bond-cni-plugin/0.log" Aug 01 01:27:45.595464 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:45.595439 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q2wgc_a4e452e0-a47b-4d14-8852-17f8567c98fc/routeoverride-cni/0.log" Aug 01 01:27:45.613138 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:45.613119 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q2wgc_a4e452e0-a47b-4d14-8852-17f8567c98fc/whereabouts-cni-bincopy/0.log" Aug 01 01:27:45.630974 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:45.630951 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q2wgc_a4e452e0-a47b-4d14-8852-17f8567c98fc/whereabouts-cni/0.log" Aug 01 01:27:45.667281 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:45.667260 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gx8gw_b09eb022-6d18-49d8-a5ba-51765459875c/kube-multus/0.log" Aug 01 01:27:45.840615 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:45.840585 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wkj56_8644d164-f68a-4c37-abd5-cc4f66cbe590/network-metrics-daemon/0.log" Aug 01 01:27:45.858390 ip-10-0-128-195 kubenswrapper[2563]: I0801 01:27:45.858365 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wkj56_8644d164-f68a-4c37-abd5-cc4f66cbe590/kube-rbac-proxy/0.log"