Jul 10 16:52:17.985697 ip-10-0-131-207 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 16:52:17.985706 ip-10-0-131-207 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 16:52:17.985713 ip-10-0-131-207 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 16:52:17.986050 ip-10-0-131-207 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 16:52:28.079449 ip-10-0-131-207 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 16:52:28.079470 ip-10-0-131-207 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fb6edf9218044b68838ae5e7d011efa7 -- Jul 10 16:54:29.992578 ip-10-0-131-207 systemd[1]: Starting Kubernetes Kubelet... Jul 10 16:54:30.407791 ip-10-0-131-207 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 16:54:30.407791 ip-10-0-131-207 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 16:54:30.407791 ip-10-0-131-207 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 16:54:30.407791 ip-10-0-131-207 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 16:54:30.407791 ip-10-0-131-207 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 16:54:30.408788 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.408698 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 16:54:30.414376 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414352 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:54:30.414376 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414372 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:54:30.414376 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414376 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:54:30.414376 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414380 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:54:30.414376 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414384 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:54:30.414376 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414387 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414391 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414394 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414396 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414399 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414402 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414404 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414407 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414409 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414412 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414415 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414417 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414420 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414423 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414425 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414428 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414430 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414433 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414435 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:54:30.414603 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414437 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414440 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414442 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414446 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414451 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414454 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414457 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414460 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414463 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414465 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414468 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414470 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414473 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414475 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414477 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414481 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414485 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414488 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414491 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:54:30.415068 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414494 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414497 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414500 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414504 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414506 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414509 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414512 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414515 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414518 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414521 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414524 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414527 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414529 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414532 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414535 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414538 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414540 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414542 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414545 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414548 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:54:30.415552 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414550 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414554 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414557 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414560 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414562 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414565 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414567 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414569 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414572 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414575 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414578 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414581 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414584 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414586 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414589 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414591 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414594 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414597 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414600 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414603 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:54:30.416039 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414607 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414609 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414612 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414615 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414617 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414620 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414624 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414627 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414629 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414632 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414634 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414637 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414639 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414642 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414644 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.414646 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415414 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415422 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415425 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415428 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:54:30.416588 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415433 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415435 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415438 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415441 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415444 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415447 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415449 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415452 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415454 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415457 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415459 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415462 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415464 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415467 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415470 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415473 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415475 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415477 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415480 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415482 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:54:30.417069 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415485 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415487 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415490 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415494 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415499 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415502 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415505 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415509 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415513 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415516 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415519 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415522 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415525 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415527 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415530 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415533 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415535 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415538 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:54:30.417591 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415541 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415543 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415546 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415548 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415550 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415553 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415556 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415559 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415562 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415565 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415568 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415570 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415573 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415575 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415578 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415580 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415583 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415585 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415588 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415591 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:54:30.418033 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415593 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415595 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415599 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415601 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415604 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415615 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415618 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415621 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415624 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415627 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415630 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415633 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415635 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415638 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415641 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415644 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415647 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415649 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415652 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415654 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:54:30.418543 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415657 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415660 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415662 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415665 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415667 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415670 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415673 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415675 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415678 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415680 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415683 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415685 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415688 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415690 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415692 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415695 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.415697 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416083 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416089 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416092 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:54:30.419041 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416094 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416097 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416100 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416102 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416105 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416108 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416110 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416113 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416115 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416118 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416120 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416122 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416125 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416128 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416130 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416133 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416135 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416138 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416142 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416145 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:54:30.419651 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416148 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416151 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416154 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416157 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416159 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416162 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416164 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416166 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416169 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416172 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416174 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416177 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416179 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416182 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416185 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416187 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416190 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416192 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416194 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416197 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:54:30.420131 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416199 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416201 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416203 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416206 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416209 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416211 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416217 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416220 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416223 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416226 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416228 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416230 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416233 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416235 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416238 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416240 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416242 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416245 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416248 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:54:30.420646 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416250 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416252 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416255 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416258 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416260 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416263 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416265 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416284 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416287 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416289 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416292 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416295 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416297 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416302 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416306 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416309 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416312 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416314 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416317 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:54:30.421167 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416322 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416326 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416329 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416331 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416334 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416336 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416339 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416342 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416344 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416347 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416349 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416352 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416354 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416357 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416359 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416361 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416364 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.416366 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416437 2567 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416445 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416451 2567 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 16:54:30.421684 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416456 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416462 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416468 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416474 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416480 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416485 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416490 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416493 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416496 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416500 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416503 2567 flags.go:64] FLAG: --cgroup-root="" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416507 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416510 2567 flags.go:64] FLAG: --client-ca-file="" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416520 2567 flags.go:64] FLAG: --cloud-provider="external" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416523 2567 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416527 2567 flags.go:64] FLAG: --cluster-domain="" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416529 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416532 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416536 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416539 2567 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416543 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416546 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416549 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 16:54:30.422177 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416552 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416555 2567 flags.go:64] FLAG: --contention-profiling="false" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416557 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416560 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416564 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416566 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416571 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416574 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416576 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416579 2567 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416582 2567 flags.go:64] FLAG: --enable-server="true" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416585 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416590 2567 flags.go:64] FLAG: --event-burst="100" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416593 2567 flags.go:64] FLAG: --event-qps="50" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416596 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416599 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416602 2567 flags.go:64] FLAG: --eviction-hard="" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416605 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416608 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416611 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416615 2567 flags.go:64] FLAG: --eviction-soft="" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416618 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416620 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416624 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416627 2567 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 16:54:30.422782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416629 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416632 2567 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416637 2567 flags.go:64] FLAG: --feature-gates="" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416641 2567 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416644 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416647 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416650 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416653 2567 flags.go:64] FLAG: --healthz-port="10248" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416656 2567 flags.go:64] FLAG: --help="false" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416659 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416662 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416665 2567 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416668 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416671 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416674 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416677 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416680 2567 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416683 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416685 2567 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416688 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416691 2567 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416693 2567 flags.go:64] FLAG: --kube-reserved="" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416696 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416699 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 16:54:30.423460 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416701 2567 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416704 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416707 2567 flags.go:64] FLAG: --lock-file="" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416711 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416713 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416716 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416721 2567 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416725 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416728 2567 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416731 2567 flags.go:64] FLAG: --logging-format="text" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416735 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416738 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416741 2567 flags.go:64] FLAG: --manifest-url="" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416744 2567 flags.go:64] FLAG: --manifest-url-header="" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416748 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416751 2567 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416755 2567 flags.go:64] FLAG: --max-pods="110" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416758 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416761 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416764 2567 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416767 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416770 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416773 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416776 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416783 2567 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 16:54:30.424039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416786 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416789 2567 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416792 2567 flags.go:64] FLAG: --pod-cidr="" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416795 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416800 2567 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416803 2567 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416806 2567 flags.go:64] FLAG: --pods-per-core="0" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416809 2567 flags.go:64] FLAG: --port="10250" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416811 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416814 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03499bfc6f79c19c0" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416818 2567 flags.go:64] FLAG: --qos-reserved="" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416821 2567 flags.go:64] FLAG: --read-only-port="10255" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416824 2567 flags.go:64] FLAG: --register-node="true" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416827 2567 flags.go:64] FLAG: --register-with-taints="" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416830 2567 flags.go:64] FLAG: --registry-burst="10" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416834 2567 flags.go:64] FLAG: --registry-qps="5" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416836 2567 flags.go:64] FLAG: --reserved-cpus="" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416839 2567 flags.go:64] FLAG: --reserved-memory="" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416844 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416847 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416850 2567 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416853 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416856 2567 flags.go:64] FLAG: --runonce="false" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416858 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416861 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 16:54:30.424665 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416864 2567 flags.go:64] FLAG: --seccomp-default="false" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416867 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416870 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416872 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416876 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416878 2567 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416881 2567 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416884 2567 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416886 2567 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416889 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416892 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416895 2567 flags.go:64] FLAG: --system-cgroups="" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416897 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416903 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416905 2567 flags.go:64] FLAG: --tls-cert-file="" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416908 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416912 2567 flags.go:64] FLAG: --tls-min-version="" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416916 2567 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416919 2567 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416921 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416924 2567 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416927 2567 flags.go:64] FLAG: --v="2" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416930 2567 flags.go:64] FLAG: --version="false" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416935 2567 flags.go:64] FLAG: --vmodule="" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416939 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 16:54:30.425263 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.416943 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417061 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417064 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417067 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417070 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417073 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417075 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417078 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417080 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417083 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417085 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417088 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417090 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417093 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417095 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417097 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417100 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417102 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417105 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417107 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:54:30.425926 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417109 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417112 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417114 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417117 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417121 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417124 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417128 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417131 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417134 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417136 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417139 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417145 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417149 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417152 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417155 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417158 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417160 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417163 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417165 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417168 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:54:30.426455 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417170 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417173 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417175 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417178 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417180 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417183 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417186 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417188 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417191 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417193 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417196 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417198 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417201 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417204 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417206 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417209 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417213 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417215 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417218 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:54:30.426968 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417222 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417225 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417228 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417230 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417233 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417238 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417240 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417243 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417245 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417248 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417250 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417253 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417255 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417258 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417260 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417264 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417285 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417288 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417291 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417294 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:54:30.427462 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417297 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417300 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417303 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417305 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417308 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417311 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417314 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417317 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417320 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417324 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417327 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417330 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417333 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417335 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417337 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417340 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417342 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417346 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417350 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:54:30.427948 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417352 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.417355 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.417990 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.425327 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.425345 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.426171 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.426178 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426237 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426242 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426246 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426249 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426252 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426254 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426257 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426260 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:54:30.428432 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426262 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426265 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426285 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426289 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426291 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426294 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426297 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426299 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426302 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426304 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426307 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426309 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426311 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426314 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426316 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426319 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426321 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426324 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426326 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426329 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:54:30.428807 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426332 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426337 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426339 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426342 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426344 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426348 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426352 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426355 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426357 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426360 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426362 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426365 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426368 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426370 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426373 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426375 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426378 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426380 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426382 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:54:30.429310 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426385 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426388 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426390 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426393 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426395 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426398 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426400 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426402 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426405 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426408 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426410 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426413 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426416 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426418 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426421 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426426 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426430 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426433 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426435 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:54:30.429768 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426438 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426441 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426443 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426446 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426448 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426450 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426453 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426455 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426458 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426460 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426463 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426465 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426468 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426471 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426473 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426476 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426478 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426481 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426483 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426486 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:54:30.430236 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426488 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426490 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426493 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426497 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426501 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426504 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426507 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426510 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426512 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426516 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426519 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426522 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426524 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.426530 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426652 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:54:30.430817 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426656 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426659 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426662 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426665 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426667 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426670 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426673 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426675 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426678 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426680 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426683 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426685 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426687 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426690 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426692 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426694 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426697 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426700 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426702 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426704 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:54:30.431189 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426707 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426709 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426711 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426714 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426717 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426719 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426722 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426726 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426729 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426732 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426734 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426737 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426739 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426742 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426744 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426746 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426749 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426751 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426754 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:54:30.431683 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426757 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426760 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426764 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426766 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426769 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426771 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426774 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426776 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426778 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426781 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426783 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426786 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426788 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426791 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426793 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426796 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426798 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426801 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426803 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:54:30.432132 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426805 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426808 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426811 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426813 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426815 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426817 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426820 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426822 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426824 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426827 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426829 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426831 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426834 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426836 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426838 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426841 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426843 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426845 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426848 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426850 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:54:30.432597 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426852 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426855 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426857 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426860 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426862 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426864 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426867 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426869 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426872 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426874 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426877 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426879 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426881 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426883 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426886 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426889 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426891 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426893 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426896 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:54:30.433074 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.426898 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:54:30.433558 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.426902 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 16:54:30.433558 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.427026 2567 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 16:54:30.433558 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.429191 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 16:54:30.433558 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.430732 2567 server.go:1019] "Starting client certificate rotation" Jul 10 16:54:30.433558 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.430834 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 16:54:30.433558 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.430875 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 16:54:30.433558 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.433377 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 16:54:30.435919 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.435857 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 16:54:30.447927 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.447900 2567 log.go:25] "Validated CRI v1 runtime API" Jul 10 16:54:30.450002 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.449984 2567 log.go:25] "Validated CRI v1 image API" Jul 10 16:54:30.450922 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.450907 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 16:54:30.456005 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.455979 2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a5e292aa-af00-46be-8f88-6f03618160d7:/dev/nvme0n1p3 bbaa32f6-c8e7-4b8e-a648-40bafebae58b:/dev/nvme0n1p4] Jul 10 16:54:30.456060 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.456006 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 16:54:30.461472 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.461355 2567 manager.go:216] Machine: {Timestamp:2026-07-10 16:54:30.459781769 +0000 UTC m=+0.347073080 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104525 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec266273408d742367d4ebe82ae36928 SystemUUID:ec266273-408d-7423-67d4-ebe82ae36928 BootID:fb6edf92-1804-4b68-838a-e5e7d011efa7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ae:f5:9d:eb:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ae:f5:9d:eb:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:90:7e:26:37:47 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 16:54:30.461472 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.461465 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 16:54:30.461592 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.461549 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 16:54:30.463049 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.463034 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 16:54:30.465764 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.465741 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 16:54:30.465918 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.465769 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-207.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 16:54:30.466528 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.466518 2567 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 16:54:30.466560 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.466532 2567 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 16:54:30.466560 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.466539 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 16:54:30.467444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.467434 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 16:54:30.468339 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.468329 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 16:54:30.471874 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.471862 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 10 16:54:30.471993 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.471983 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 16:54:30.472028 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.472014 2567 kubelet.go:486] "Attempting to sync node with API server" Jul 10 16:54:30.472055 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.472028 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 16:54:30.472055 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.472043 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 16:54:30.472055 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.472052 2567 kubelet.go:398] "Adding apiserver pod source" Jul 10 16:54:30.472128 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.472068 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 16:54:30.472462 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.472449 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 16:54:30.472499 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.472471 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 16:54:30.474680 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.474664 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 16:54:30.476116 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.476101 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 16:54:30.476180 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.476133 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 16:54:30.477325 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.477313 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 16:54:30.477380 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.477331 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 16:54:30.477380 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.477337 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 16:54:30.477380 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.477343 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 16:54:30.477380 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.477351 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 16:54:30.477380 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.477357 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 16:54:30.477380 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.477367 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 16:54:30.477380 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.477375 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 16:54:30.477380 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.477383 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 16:54:30.477598 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.477388 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 16:54:30.477598 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.477397 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 16:54:30.477598 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.477402 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 16:54:30.478302 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.478291 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 16:54:30.478302 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.478302 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 16:54:30.481638 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.481624 2567 server.go:1268] "Started kubelet" Jul 10 16:54:30.481753 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.481722 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 16:54:30.482353 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.481936 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 16:54:30.482658 ip-10-0-131-207 systemd[1]: Started Kubernetes Kubelet. Jul 10 16:54:30.486347 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.486286 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 16:54:30.487818 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.487777 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 16:54:30.489218 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.489196 2567 server.go:310] "Adding debug handlers to kubelet server" Jul 10 16:54:30.489597 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.489581 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-207.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 16:54:30.490214 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.490092 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-207.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 16:54:30.490214 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.490093 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 16:54:30.491338 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.491310 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 16:54:30.493368 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.493349 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 16:54:30.493368 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.493358 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 16:54:30.493512 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.493469 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:30.493568 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.493525 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6h7f" Jul 10 16:54:30.493810 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.492843 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-207.ec2.internal.18c0fbf24a0ad430 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-207.ec2.internal,UID:ip-10-0-131-207.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-207.ec2.internal,},FirstTimestamp:2026-07-10 16:54:30.481597488 +0000 UTC m=+0.368888799,LastTimestamp:2026-07-10 16:54:30.481597488 +0000 UTC m=+0.368888799,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-207.ec2.internal,}" Jul 10 16:54:30.493957 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.493942 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 16:54:30.494052 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.494042 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 16:54:30.494106 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.493967 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 16:54:30.494171 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.494158 2567 reconstruct.go:97] "Volume reconstruction finished" Jul 10 16:54:30.494218 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.494172 2567 reconciler.go:29] "Reconciler: start to sync state" Jul 10 16:54:30.494549 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.494530 2567 factory.go:55] Registering systemd factory Jul 10 16:54:30.494625 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.494551 2567 factory.go:223] Registration of the systemd container factory successfully Jul 10 16:54:30.494754 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.494740 2567 factory.go:153] Registering CRI-O factory Jul 10 16:54:30.494794 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.494756 2567 factory.go:223] Registration of the crio container factory successfully Jul 10 16:54:30.494822 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.494798 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 16:54:30.494822 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.494819 2567 factory.go:103] Registering Raw factory Jul 10 16:54:30.494880 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.494835 2567 manager.go:1196] Started watching for new ooms in manager Jul 10 16:54:30.495257 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.495242 2567 manager.go:349] Starting recovery of all containers Jul 10 16:54:30.502083 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.502055 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-207.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 16:54:30.502219 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.502198 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 16:54:30.503017 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.502823 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6h7f" Jul 10 16:54:30.504979 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:30.504958 2567 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service/pids.max": read /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service/pids.max: no such device Jul 10 16:54:30.505435 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.505422 2567 manager.go:354] Recovery completed Jul 10 16:54:30.509597 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.509585 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:30.511440 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.511425 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:30.511519 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.511456 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:30.511519 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.511467 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:30.511962 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.511933 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 16:54:30.511962 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.511946 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 16:54:30.511962 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.511959 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:30.512149 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.511960 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 10 16:54:30.512727 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.512713 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:30.512808 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.512741 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:30.512808 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.512751 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:30.513471 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.513460 2567 policy_none.go:49] "None policy: Start" Jul 10 16:54:30.513507 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.513478 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 16:54:30.513507 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.513489 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 16:54:30.515023 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.515012 2567 policy_none.go:47] "Start" Jul 10 16:54:30.548828 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.548810 2567 manager.go:337] "Starting Device Plugin manager" Jul 10 16:54:30.548928 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.548843 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 16:54:30.548928 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.548853 2567 server.go:87] "Starting device plugin registration server" Jul 10 16:54:30.549122 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.549107 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 16:54:30.549167 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.549127 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 16:54:30.549222 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.549205 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 16:54:30.549330 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.549311 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 16:54:30.549330 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.549325 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 16:54:30.549811 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.549790 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 16:54:30.549885 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.549844 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:30.649784 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.649756 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:30.651362 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.650719 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:30.651362 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.650754 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:30.651362 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.650764 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:30.651362 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.650792 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.662345 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.662296 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.662345 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.662325 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-207.ec2.internal\": node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:30.673877 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.673851 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 16:54:30.675176 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.675155 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 16:54:30.675285 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.675200 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 16:54:30.675285 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.675222 2567 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 16:54:30.675285 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.675261 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 16:54:30.678151 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.678134 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:54:30.683721 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.683707 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:30.776109 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.776073 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-207.ec2.internal"] Jul 10 16:54:30.776332 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.776187 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:30.777982 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.777950 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:30.778108 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.777993 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:30.778108 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.778004 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:30.779534 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.779520 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:30.779681 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.779650 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.779736 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.779690 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:30.780320 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.780303 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:30.780320 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.780318 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:30.780439 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.780334 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:30.780439 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.780339 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:30.780439 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.780344 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:30.780439 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.780353 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:30.781501 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.781489 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.781553 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.781515 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:30.783758 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.783743 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:30.784006 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.783992 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:30.784053 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.784019 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:30.784053 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.784029 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:30.796022 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.795994 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b203731d72488e0ccf24b1f1be91c047-config\") pod \"kube-apiserver-proxy-ip-10-0-131-207.ec2.internal\" (UID: \"b203731d72488e0ccf24b1f1be91c047\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.796022 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.796022 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27c9d0fc09f37a4d51b4693af721e488-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal\" (UID: \"27c9d0fc09f37a4d51b4693af721e488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.796144 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.796039 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27c9d0fc09f37a4d51b4693af721e488-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal\" (UID: \"27c9d0fc09f37a4d51b4693af721e488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.803995 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.803972 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-207.ec2.internal\" not found" node="ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.808295 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.808263 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-207.ec2.internal\" not found" node="ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.884325 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.884289 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:30.896700 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.896671 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27c9d0fc09f37a4d51b4693af721e488-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal\" (UID: \"27c9d0fc09f37a4d51b4693af721e488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.896816 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.896704 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27c9d0fc09f37a4d51b4693af721e488-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal\" (UID: \"27c9d0fc09f37a4d51b4693af721e488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.896816 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.896722 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b203731d72488e0ccf24b1f1be91c047-config\") pod \"kube-apiserver-proxy-ip-10-0-131-207.ec2.internal\" (UID: \"b203731d72488e0ccf24b1f1be91c047\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.896816 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.896774 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b203731d72488e0ccf24b1f1be91c047-config\") pod \"kube-apiserver-proxy-ip-10-0-131-207.ec2.internal\" (UID: \"b203731d72488e0ccf24b1f1be91c047\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.896816 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.896781 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27c9d0fc09f37a4d51b4693af721e488-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal\" (UID: \"27c9d0fc09f37a4d51b4693af721e488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.896816 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:30.896784 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27c9d0fc09f37a4d51b4693af721e488-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal\" (UID: \"27c9d0fc09f37a4d51b4693af721e488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal" Jul 10 16:54:30.985457 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:30.985378 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:31.089846 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:31.089793 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:31.107523 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:31.107493 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal" Jul 10 16:54:31.110916 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:31.110884 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-207.ec2.internal" Jul 10 16:54:31.191851 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:31.191801 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:31.293965 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:31.293878 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:31.316298 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:31.316267 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:54:31.397119 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:31.397082 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:31.430988 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:31.430960 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 16:54:31.431597 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:31.431136 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 16:54:31.431597 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:31.431174 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 16:54:31.497450 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:31.497418 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:31.508227 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:31.508188 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 16:49:31 +0000 UTC" deadline="2028-01-03 17:27:06.954264306 +0000 UTC" Jul 10 16:54:31.508227 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:31.508226 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13008h32m35.446042946s" Jul 10 16:54:31.598552 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:31.598480 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:31.698586 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:31.698547 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:31.731217 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:31.731182 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb203731d72488e0ccf24b1f1be91c047.slice/crio-0540f5c545a38a04e5884dc76e13d4a3d5e2ed13cf9699ac758e9a58d4edd40b WatchSource:0}: Error finding container 0540f5c545a38a04e5884dc76e13d4a3d5e2ed13cf9699ac758e9a58d4edd40b: Status 404 returned error can't find the container with id 0540f5c545a38a04e5884dc76e13d4a3d5e2ed13cf9699ac758e9a58d4edd40b Jul 10 16:54:31.731685 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:31.731666 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c9d0fc09f37a4d51b4693af721e488.slice/crio-187fa11cb7e9b42b09390bd139046335284186d26036b142f65451e17fdb14a9 WatchSource:0}: Error finding container 187fa11cb7e9b42b09390bd139046335284186d26036b142f65451e17fdb14a9: Status 404 returned error can't find the container with id 187fa11cb7e9b42b09390bd139046335284186d26036b142f65451e17fdb14a9 Jul 10 16:54:31.738646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:31.738626 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:54:31.783467 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:31.783441 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:54:31.799630 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:31.799613 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-207.ec2.internal\" not found" Jul 10 16:54:31.896718 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:31.896657 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:54:31.995189 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:31.995138 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal" Jul 10 16:54:32.006494 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.006466 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 16:54:32.007581 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.007566 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-207.ec2.internal" Jul 10 16:54:32.014412 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.014393 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 16:54:32.360559 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.360478 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:54:32.474720 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.474687 2567 apiserver.go:52] "Watching apiserver" Jul 10 16:54:32.485353 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.484929 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 16:54:32.487522 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.487488 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-rf7b9","kube-system/kube-apiserver-proxy-ip-10-0-131-207.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg","openshift-image-registry/node-ca-g924d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal","openshift-multus/multus-d62w2","openshift-network-diagnostics/network-check-target-8gqbc","openshift-network-operator/iptables-alerter-rnvf6","kube-system/konnectivity-agent-n8qpc","openshift-cluster-node-tuning-operator/tuned-5jgps","openshift-dns/node-resolver-kzbjz","openshift-multus/multus-additional-cni-plugins-4blh9","openshift-multus/network-metrics-daemon-6cldf","openshift-ovn-kubernetes/ovnkube-node-trj2g"] Jul 10 16:54:32.489324 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.489297 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:32.489830 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:32.489787 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:32.490749 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.490729 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kzbjz" Jul 10 16:54:32.492393 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.492130 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.496416 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.493445 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g924d" Jul 10 16:54:32.496416 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.493611 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 16:54:32.496416 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.495148 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-94qtp\"" Jul 10 16:54:32.496416 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.495148 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 16:54:32.496416 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.495440 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 16:54:32.496416 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.495440 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 16:54:32.496416 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.495792 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.497229 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.497209 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.498582 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.498498 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 16:54:32.498582 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.498530 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 16:54:32.498582 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.498556 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 16:54:32.498785 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.498593 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vrzvb\"" Jul 10 16:54:32.498785 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.498505 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-stnzf\"" Jul 10 16:54:32.498785 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.498749 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n8qpc" Jul 10 16:54:32.498943 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.498805 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 16:54:32.499091 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.499074 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 16:54:32.499791 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.499762 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:32.499893 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:32.499828 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:32.501203 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.501182 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 16:54:32.502101 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.501675 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rnvf6" Jul 10 16:54:32.503659 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.503290 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.504781 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.504748 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:32.504879 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:32.504828 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:32.505034 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.505014 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 16:54:32.505292 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.505263 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 16:54:32.505400 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.505309 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 16:54:32.505475 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.505449 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 16:54:32.505533 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.505470 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 16:54:32.505619 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.505571 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 16:54:32.505708 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.505687 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 16:54:32.505833 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.505717 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f8qx5\"" Jul 10 16:54:32.506000 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.505980 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jdbq9\"" Jul 10 16:54:32.507527 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.506511 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 16:54:32.507527 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.506625 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 16:54:32.507527 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.506854 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:54:32.507527 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.507049 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 16:54:32.507527 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.507089 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4fz8\"" Jul 10 16:54:32.507527 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.507144 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6jdbp\"" Jul 10 16:54:32.507527 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.507315 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-chnvg\"" Jul 10 16:54:32.507527 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.507372 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 16:54:32.507527 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.507430 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 16:54:32.508043 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.507674 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 16:54:32.508583 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508385 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 16:54:32.508583 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508486 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/441cb5b6-d258-4ccc-a950-62b1179f6c1a-agent-certs\") pod \"konnectivity-agent-n8qpc\" (UID: \"441cb5b6-d258-4ccc-a950-62b1179f6c1a\") " pod="kube-system/konnectivity-agent-n8qpc" Jul 10 16:54:32.508583 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508519 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-systemd-units\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.508583 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508545 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-run-netns\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.508583 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508571 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-log-socket\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.508887 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508596 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-system-cni-dir\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.508887 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508665 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-var-lib-cni-bin\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.508887 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508729 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs\") pod \"network-metrics-daemon-6cldf\" (UID: \"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c\") " pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:32.508887 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508767 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-env-overrides\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.508887 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508788 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rb94\" (UniqueName: \"kubernetes.io/projected/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-kube-api-access-4rb94\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.508887 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508804 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-run-netns\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.508887 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508872 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-registration-dir\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.509252 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508924 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srnsf\" (UniqueName: \"kubernetes.io/projected/5ac53225-b3f6-4a01-a823-03c876fdc239-kube-api-access-srnsf\") pod \"node-ca-g924d\" (UID: \"5ac53225-b3f6-4a01-a823-03c876fdc239\") " pod="openshift-image-registry/node-ca-g924d" Jul 10 16:54:32.509252 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.508973 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-ovnkube-config\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.509252 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509007 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/403bed03-bca0-461d-9cbf-767071e7a527-multus-daemon-config\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.509252 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509053 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-etc-kubernetes\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.509252 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509086 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l5kn\" (UniqueName: \"kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn\") pod \"network-check-target-8gqbc\" (UID: \"832da8b4-c715-4416-9881-77c2baa30314\") " pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:32.509252 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509122 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-sys-fs\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.509252 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509163 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-var-lib-openvswitch\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.509252 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509189 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.509252 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509216 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff-hosts-file\") pod \"node-resolver-kzbjz\" (UID: \"dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff\") " pod="openshift-dns/node-resolver-kzbjz" Jul 10 16:54:32.509252 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509242 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff-tmp-dir\") pod \"node-resolver-kzbjz\" (UID: \"dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff\") " pod="openshift-dns/node-resolver-kzbjz" Jul 10 16:54:32.509603 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-run-ovn\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.509603 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509284 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:54:32.509603 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509317 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-var-lib-kubelet\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.509603 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509452 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.509603 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509480 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-ovnkube-script-lib\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.509603 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509527 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-multus-socket-dir-parent\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.509603 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509559 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-run-multus-certs\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.509603 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509596 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/441cb5b6-d258-4ccc-a950-62b1179f6c1a-konnectivity-ca\") pod \"konnectivity-agent-n8qpc\" (UID: \"441cb5b6-d258-4ccc-a950-62b1179f6c1a\") " pod="kube-system/konnectivity-agent-n8qpc" Jul 10 16:54:32.509883 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509628 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/403bed03-bca0-461d-9cbf-767071e7a527-cni-binary-copy\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.509883 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509652 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-multus-conf-dir\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.509883 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509690 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4gq\" (UniqueName: \"kubernetes.io/projected/403bed03-bca0-461d-9cbf-767071e7a527-kube-api-access-cr4gq\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.509883 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509718 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.509883 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509734 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ac53225-b3f6-4a01-a823-03c876fdc239-host\") pod \"node-ca-g924d\" (UID: \"5ac53225-b3f6-4a01-a823-03c876fdc239\") " pod="openshift-image-registry/node-ca-g924d" Jul 10 16:54:32.509883 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-cnibin\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.509883 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509790 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-os-release\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.509883 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509815 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx9xb\" (UniqueName: \"kubernetes.io/projected/e54f7489-3734-4c1e-909f-bcee0f6f8ced-kube-api-access-jx9xb\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.509883 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509855 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-ovn-node-metrics-cert\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.510141 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509904 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-multus-cni-dir\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.510141 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509929 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-socket-dir\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.510141 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.509975 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.510141 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510003 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ac53225-b3f6-4a01-a823-03c876fdc239-serviceca\") pod \"node-ca-g924d\" (UID: \"5ac53225-b3f6-4a01-a823-03c876fdc239\") " pod="openshift-image-registry/node-ca-g924d" Jul 10 16:54:32.510141 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510041 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-kubelet\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.510141 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510072 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-slash\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.510141 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510097 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-cni-netd\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.510377 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510143 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv9wn\" (UniqueName: \"kubernetes.io/projected/dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff-kube-api-access-bv9wn\") pod \"node-resolver-kzbjz\" (UID: \"dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff\") " pod="openshift-dns/node-resolver-kzbjz" Jul 10 16:54:32.510377 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510173 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-hostroot\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.510377 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510202 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-run-systemd\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.510377 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510236 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-etc-openvswitch\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.510377 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510282 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-run-k8s-cni-cncf-io\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.510377 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510312 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g29dv\" (UniqueName: \"kubernetes.io/projected/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-kube-api-access-g29dv\") pod \"network-metrics-daemon-6cldf\" (UID: \"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c\") " pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:32.510377 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510336 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-device-dir\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.510610 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510408 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.510610 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510443 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-var-lib-cni-multus\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.510610 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510469 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-run-openvswitch\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.510610 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510495 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-node-log\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.510610 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.510525 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-cni-bin\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.519045 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.518986 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 16:54:32.519431 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.519414 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzq62\"" Jul 10 16:54:32.521471 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.521432 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 16:54:32.525558 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.524664 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 16:54:32.550892 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.550862 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ldwf2" Jul 10 16:54:32.560199 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.560168 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ldwf2" Jul 10 16:54:32.595340 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.595314 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 16:54:32.611103 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611021 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-registration-dir\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.611103 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611068 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srnsf\" (UniqueName: \"kubernetes.io/projected/5ac53225-b3f6-4a01-a823-03c876fdc239-kube-api-access-srnsf\") pod \"node-ca-g924d\" (UID: \"5ac53225-b3f6-4a01-a823-03c876fdc239\") " pod="openshift-image-registry/node-ca-g924d" Jul 10 16:54:32.611103 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611097 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-ovnkube-config\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.611369 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611126 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/403bed03-bca0-461d-9cbf-767071e7a527-multus-daemon-config\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.611369 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611153 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-etc-kubernetes\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.611369 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611184 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-system-cni-dir\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.611369 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611215 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-kubernetes\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.611369 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611239 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-var-lib-kubelet\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.611369 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5kn\" (UniqueName: \"kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn\") pod \"network-check-target-8gqbc\" (UID: \"832da8b4-c715-4416-9881-77c2baa30314\") " pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:32.611369 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611289 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-sys-fs\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.611369 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611335 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-var-lib-openvswitch\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.611369 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611364 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611394 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff-hosts-file\") pod \"node-resolver-kzbjz\" (UID: \"dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff\") " pod="openshift-dns/node-resolver-kzbjz" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611421 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff-tmp-dir\") pod \"node-resolver-kzbjz\" (UID: \"dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff\") " pod="openshift-dns/node-resolver-kzbjz" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611452 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-sysctl-conf\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611479 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f36b025e-33b6-4b94-8b11-743ee8c1ebec-host-slash\") pod \"iptables-alerter-rnvf6\" (UID: \"f36b025e-33b6-4b94-8b11-743ee8c1ebec\") " pod="openshift-network-operator/iptables-alerter-rnvf6" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611505 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-lib-modules\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611537 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-dbus\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611564 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-run-ovn\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611589 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-var-lib-kubelet\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611619 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611647 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-ovnkube-script-lib\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611672 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-multus-socket-dir-parent\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611697 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-run-multus-certs\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611721 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-cnibin\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611744 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-host\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611747 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-registration-dir\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611783 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/441cb5b6-d258-4ccc-a950-62b1179f6c1a-konnectivity-ca\") pod \"konnectivity-agent-n8qpc\" (UID: \"441cb5b6-d258-4ccc-a950-62b1179f6c1a\") " pod="kube-system/konnectivity-agent-n8qpc" Jul 10 16:54:32.611862 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611809 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/403bed03-bca0-461d-9cbf-767071e7a527-cni-binary-copy\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611870 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-multus-conf-dir\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611921 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4gq\" (UniqueName: \"kubernetes.io/projected/403bed03-bca0-461d-9cbf-767071e7a527-kube-api-access-cr4gq\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611948 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-sys\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.611994 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-kubelet-config\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612021 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ac53225-b3f6-4a01-a823-03c876fdc239-host\") pod \"node-ca-g924d\" (UID: \"5ac53225-b3f6-4a01-a823-03c876fdc239\") " pod="openshift-image-registry/node-ca-g924d" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612046 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-cnibin\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612070 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-os-release\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612097 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612145 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r88b2\" (UniqueName: \"kubernetes.io/projected/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-kube-api-access-r88b2\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612170 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-tuned\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612192 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8a37165-0d14-486d-8ff2-bdbdf1818049-tmp\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612251 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx9xb\" (UniqueName: \"kubernetes.io/projected/e54f7489-3734-4c1e-909f-bcee0f6f8ced-kube-api-access-jx9xb\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612280 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-ovn-node-metrics-cert\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612304 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-multus-cni-dir\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612331 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612375 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdhh7\" (UniqueName: \"kubernetes.io/projected/f36b025e-33b6-4b94-8b11-743ee8c1ebec-kube-api-access-mdhh7\") pod \"iptables-alerter-rnvf6\" (UID: \"f36b025e-33b6-4b94-8b11-743ee8c1ebec\") " pod="openshift-network-operator/iptables-alerter-rnvf6" Jul 10 16:54:32.612646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612409 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-sysctl-d\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612451 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smjrp\" (UniqueName: \"kubernetes.io/projected/b8a37165-0d14-486d-8ff2-bdbdf1818049-kube-api-access-smjrp\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612480 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-socket-dir\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612506 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612532 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ac53225-b3f6-4a01-a823-03c876fdc239-serviceca\") pod \"node-ca-g924d\" (UID: \"5ac53225-b3f6-4a01-a823-03c876fdc239\") " pod="openshift-image-registry/node-ca-g924d" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612559 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-kubelet\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612587 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-slash\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612614 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-cni-netd\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612640 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv9wn\" (UniqueName: \"kubernetes.io/projected/dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff-kube-api-access-bv9wn\") pod \"node-resolver-kzbjz\" (UID: \"dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff\") " pod="openshift-dns/node-resolver-kzbjz" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612671 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-hostroot\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612720 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612722 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-run-systemd\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612773 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-run-systemd\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612793 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-etc-openvswitch\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612830 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-run-k8s-cni-cncf-io\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.612926 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-var-lib-openvswitch\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.613034 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-etc-openvswitch\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.613469 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.613127 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff-hosts-file\") pod \"node-resolver-kzbjz\" (UID: \"dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff\") " pod="openshift-dns/node-resolver-kzbjz" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.613245 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/403bed03-bca0-461d-9cbf-767071e7a527-multus-daemon-config\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.613487 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-multus-socket-dir-parent\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.613506 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-ovnkube-config\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.613542 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-run-ovn\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.613582 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-run-k8s-cni-cncf-io\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.613593 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff-tmp-dir\") pod \"node-resolver-kzbjz\" (UID: \"dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff\") " pod="openshift-dns/node-resolver-kzbjz" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.613629 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ac53225-b3f6-4a01-a823-03c876fdc239-host\") pod \"node-ca-g924d\" (UID: \"5ac53225-b3f6-4a01-a823-03c876fdc239\") " pod="openshift-image-registry/node-ca-g924d" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.613668 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f36b025e-33b6-4b94-8b11-743ee8c1ebec-iptables-alerter-script\") pod \"iptables-alerter-rnvf6\" (UID: \"f36b025e-33b6-4b94-8b11-743ee8c1ebec\") " pod="openshift-network-operator/iptables-alerter-rnvf6" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.613720 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-cnibin\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.613748 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-var-lib-kubelet\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.613778 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.614121 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-ovnkube-script-lib\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.614179 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-os-release\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.614249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.614205 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-run-multus-certs\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.614903 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.614494 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-multus-cni-dir\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.614903 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.614503 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 16:54:32.614903 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.614688 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.614903 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.614885 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-socket-dir\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.615098 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.614935 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-cni-netd\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.615098 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.614975 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-slash\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.615209 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.615146 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ac53225-b3f6-4a01-a823-03c876fdc239-serviceca\") pod \"node-ca-g924d\" (UID: \"5ac53225-b3f6-4a01-a823-03c876fdc239\") " pod="openshift-image-registry/node-ca-g924d" Jul 10 16:54:32.615209 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.615134 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-kubelet\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.615318 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.615228 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-multus-conf-dir\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.615559 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.615535 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-sys-fs\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.615645 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.615598 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-modprobe-d\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.615645 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.615591 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-hostroot\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.615645 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.615629 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-etc-kubernetes\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.615819 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.615679 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-systemd\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.615819 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.615710 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:32.615819 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.615768 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g29dv\" (UniqueName: \"kubernetes.io/projected/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-kube-api-access-g29dv\") pod \"network-metrics-daemon-6cldf\" (UID: \"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c\") " pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:32.615819 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.615797 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/403bed03-bca0-461d-9cbf-767071e7a527-cni-binary-copy\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.616010 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.615949 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-device-dir\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.616010 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.615992 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.616122 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616043 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e54f7489-3734-4c1e-909f-bcee0f6f8ced-device-dir\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.616122 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616093 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-var-lib-cni-multus\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.616122 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616099 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.616290 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616133 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-run-openvswitch\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.616290 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616157 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-node-log\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.616290 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616211 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-run-openvswitch\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.616290 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616225 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-var-lib-cni-multus\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.616290 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616250 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-cni-bin\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.616646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616307 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-cni-bin\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.616646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616311 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-node-log\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.616646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616349 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-sysconfig\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.616646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616381 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/441cb5b6-d258-4ccc-a950-62b1179f6c1a-agent-certs\") pod \"konnectivity-agent-n8qpc\" (UID: \"441cb5b6-d258-4ccc-a950-62b1179f6c1a\") " pod="kube-system/konnectivity-agent-n8qpc" Jul 10 16:54:32.616646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-systemd-units\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.616646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616562 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-run-netns\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.616646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616579 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-systemd-units\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.616646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616611 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-log-socket\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.616646 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616630 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-system-cni-dir\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.617115 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616667 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-var-lib-cni-bin\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.617115 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616677 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-host-run-netns\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.617115 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616697 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-os-release\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.617115 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616727 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-system-cni-dir\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.617115 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616767 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-cni-binary-copy\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.617115 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616775 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-var-lib-cni-bin\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.617115 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616813 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs\") pod \"network-metrics-daemon-6cldf\" (UID: \"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c\") " pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:32.617115 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616835 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-log-socket\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.617115 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616864 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-env-overrides\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.617115 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616883 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rb94\" (UniqueName: \"kubernetes.io/projected/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-kube-api-access-4rb94\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.617115 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.616906 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-run-netns\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.617115 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:32.617023 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:32.617654 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:32.617157 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs podName:f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c nodeName:}" failed. No retries permitted until 2026-07-10 16:54:33.117135565 +0000 UTC m=+3.004426885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs") pod "network-metrics-daemon-6cldf" (UID: "f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:32.617654 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.617221 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.617654 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.617255 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-run\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.617654 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.617341 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/403bed03-bca0-461d-9cbf-767071e7a527-host-run-netns\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.617869 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.617684 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/441cb5b6-d258-4ccc-a950-62b1179f6c1a-konnectivity-ca\") pod \"konnectivity-agent-n8qpc\" (UID: \"441cb5b6-d258-4ccc-a950-62b1179f6c1a\") " pod="kube-system/konnectivity-agent-n8qpc" Jul 10 16:54:32.617869 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.617732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-env-overrides\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.621420 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:32.621394 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:54:32.621420 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:32.621421 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:54:32.621609 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:32.621437 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l5kn for pod openshift-network-diagnostics/network-check-target-8gqbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:32.621609 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:32.621508 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn podName:832da8b4-c715-4416-9881-77c2baa30314 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:33.121487769 +0000 UTC m=+3.008779094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5kn" (UniqueName: "kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn") pod "network-check-target-8gqbc" (UID: "832da8b4-c715-4416-9881-77c2baa30314") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:32.621803 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.621663 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/441cb5b6-d258-4ccc-a950-62b1179f6c1a-agent-certs\") pod \"konnectivity-agent-n8qpc\" (UID: \"441cb5b6-d258-4ccc-a950-62b1179f6c1a\") " pod="kube-system/konnectivity-agent-n8qpc" Jul 10 16:54:32.622359 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.622309 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4gq\" (UniqueName: \"kubernetes.io/projected/403bed03-bca0-461d-9cbf-767071e7a527-kube-api-access-cr4gq\") pod \"multus-d62w2\" (UID: \"403bed03-bca0-461d-9cbf-767071e7a527\") " pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.625580 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.625552 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g29dv\" (UniqueName: \"kubernetes.io/projected/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-kube-api-access-g29dv\") pod \"network-metrics-daemon-6cldf\" (UID: \"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c\") " pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:32.625832 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.625808 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srnsf\" (UniqueName: \"kubernetes.io/projected/5ac53225-b3f6-4a01-a823-03c876fdc239-kube-api-access-srnsf\") pod \"node-ca-g924d\" (UID: \"5ac53225-b3f6-4a01-a823-03c876fdc239\") " pod="openshift-image-registry/node-ca-g924d" Jul 10 16:54:32.626839 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.626814 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx9xb\" (UniqueName: \"kubernetes.io/projected/e54f7489-3734-4c1e-909f-bcee0f6f8ced-kube-api-access-jx9xb\") pod \"aws-ebs-csi-driver-node-mxnqg\" (UID: \"e54f7489-3734-4c1e-909f-bcee0f6f8ced\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.627129 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.627107 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-ovn-node-metrics-cert\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.628848 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.628824 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv9wn\" (UniqueName: \"kubernetes.io/projected/dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff-kube-api-access-bv9wn\") pod \"node-resolver-kzbjz\" (UID: \"dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff\") " pod="openshift-dns/node-resolver-kzbjz" Jul 10 16:54:32.631414 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.631391 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rb94\" (UniqueName: \"kubernetes.io/projected/da59a0ce-b509-4f6e-8685-b25d58a3fc2d-kube-api-access-4rb94\") pod \"ovnkube-node-trj2g\" (UID: \"da59a0ce-b509-4f6e-8685-b25d58a3fc2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.683470 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.683414 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-207.ec2.internal" event={"ID":"b203731d72488e0ccf24b1f1be91c047","Type":"ContainerStarted","Data":"0540f5c545a38a04e5884dc76e13d4a3d5e2ed13cf9699ac758e9a58d4edd40b"} Jul 10 16:54:32.685151 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.685121 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal" event={"ID":"27c9d0fc09f37a4d51b4693af721e488","Type":"ContainerStarted","Data":"187fa11cb7e9b42b09390bd139046335284186d26036b142f65451e17fdb14a9"} Jul 10 16:54:32.718333 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718273 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-sysconfig\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.718333 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718346 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-os-release\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.718598 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718378 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-cni-binary-copy\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.718598 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718446 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-sysconfig\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.718598 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718456 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.718598 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718509 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-run\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.718598 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718540 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-system-cni-dir\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.718598 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718568 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-kubernetes\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.718598 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718593 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-var-lib-kubelet\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.718969 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718637 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-sysctl-conf\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.718969 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718666 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f36b025e-33b6-4b94-8b11-743ee8c1ebec-host-slash\") pod \"iptables-alerter-rnvf6\" (UID: \"f36b025e-33b6-4b94-8b11-743ee8c1ebec\") " pod="openshift-network-operator/iptables-alerter-rnvf6" Jul 10 16:54:32.718969 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718692 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-lib-modules\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.718969 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718716 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-dbus\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:32.718969 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718751 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-cnibin\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.718969 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718774 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-host\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.718969 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718823 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-sys\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.718969 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718848 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-kubelet-config\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:32.718969 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718877 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.718969 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718906 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r88b2\" (UniqueName: \"kubernetes.io/projected/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-kube-api-access-r88b2\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.718969 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718932 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-tuned\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.718969 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718957 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8a37165-0d14-486d-8ff2-bdbdf1818049-tmp\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.718988 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719015 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdhh7\" (UniqueName: \"kubernetes.io/projected/f36b025e-33b6-4b94-8b11-743ee8c1ebec-kube-api-access-mdhh7\") pod \"iptables-alerter-rnvf6\" (UID: \"f36b025e-33b6-4b94-8b11-743ee8c1ebec\") " pod="openshift-network-operator/iptables-alerter-rnvf6" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719040 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-sysctl-d\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719048 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719066 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smjrp\" (UniqueName: \"kubernetes.io/projected/b8a37165-0d14-486d-8ff2-bdbdf1818049-kube-api-access-smjrp\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719051 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-cni-binary-copy\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719114 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-os-release\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719137 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f36b025e-33b6-4b94-8b11-743ee8c1ebec-iptables-alerter-script\") pod \"iptables-alerter-rnvf6\" (UID: \"f36b025e-33b6-4b94-8b11-743ee8c1ebec\") " pod="openshift-network-operator/iptables-alerter-rnvf6" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719171 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-modprobe-d\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719178 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-run\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719184 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-sys\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719200 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-systemd\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719228 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719233 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-system-cni-dir\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719424 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-dbus\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:32.719589 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719562 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-lib-modules\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719617 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-cnibin\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719672 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-host\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719789 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719844 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f36b025e-33b6-4b94-8b11-743ee8c1ebec-iptables-alerter-script\") pod \"iptables-alerter-rnvf6\" (UID: \"f36b025e-33b6-4b94-8b11-743ee8c1ebec\") " pod="openshift-network-operator/iptables-alerter-rnvf6" Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719854 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-kubelet-config\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719906 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.719965 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-modprobe-d\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.720019 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-var-lib-kubelet\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.720091 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-sysctl-conf\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.720276 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-kubernetes\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:32.720338 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.720394 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-systemd\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.720400 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-sysctl-d\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.720444 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:32.720449 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret podName:e3390d32-503c-4e8f-ab8d-c1b3881f2df9 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:33.220424325 +0000 UTC m=+3.107715625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret") pod "global-pull-secret-syncer-rf7b9" (UID: "e3390d32-503c-4e8f-ab8d-c1b3881f2df9") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:32.721030 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.720468 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f36b025e-33b6-4b94-8b11-743ee8c1ebec-host-slash\") pod \"iptables-alerter-rnvf6\" (UID: \"f36b025e-33b6-4b94-8b11-743ee8c1ebec\") " pod="openshift-network-operator/iptables-alerter-rnvf6" Jul 10 16:54:32.723298 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.723259 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8a37165-0d14-486d-8ff2-bdbdf1818049-tmp\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.723410 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.723309 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b8a37165-0d14-486d-8ff2-bdbdf1818049-etc-tuned\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.733491 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.733457 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdhh7\" (UniqueName: \"kubernetes.io/projected/f36b025e-33b6-4b94-8b11-743ee8c1ebec-kube-api-access-mdhh7\") pod \"iptables-alerter-rnvf6\" (UID: \"f36b025e-33b6-4b94-8b11-743ee8c1ebec\") " pod="openshift-network-operator/iptables-alerter-rnvf6" Jul 10 16:54:32.734057 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.734029 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r88b2\" (UniqueName: \"kubernetes.io/projected/b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc-kube-api-access-r88b2\") pod \"multus-additional-cni-plugins-4blh9\" (UID: \"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc\") " pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.734201 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.734036 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smjrp\" (UniqueName: \"kubernetes.io/projected/b8a37165-0d14-486d-8ff2-bdbdf1818049-kube-api-access-smjrp\") pod \"tuned-5jgps\" (UID: \"b8a37165-0d14-486d-8ff2-bdbdf1818049\") " pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:32.808531 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.808490 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kzbjz" Jul 10 16:54:32.826439 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.826407 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" Jul 10 16:54:32.836778 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.836752 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4blh9" Jul 10 16:54:32.843181 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.843155 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:32.852386 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.852354 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d62w2" Jul 10 16:54:32.859298 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.859256 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n8qpc" Jul 10 16:54:32.870277 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.870224 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g924d" Jul 10 16:54:32.880249 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.880226 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rnvf6" Jul 10 16:54:32.894627 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:32.894597 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5jgps" Jul 10 16:54:33.123011 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:33.122920 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs\") pod \"network-metrics-daemon-6cldf\" (UID: \"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c\") " pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:33.123011 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:33.122974 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5kn\" (UniqueName: \"kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn\") pod \"network-check-target-8gqbc\" (UID: \"832da8b4-c715-4416-9881-77c2baa30314\") " pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:33.123248 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:33.123124 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:33.123248 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:33.123147 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:54:33.123248 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:33.123169 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:54:33.123248 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:33.123183 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l5kn for pod openshift-network-diagnostics/network-check-target-8gqbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:33.123248 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:33.123199 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs podName:f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c nodeName:}" failed. No retries permitted until 2026-07-10 16:54:34.123179134 +0000 UTC m=+4.010470456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs") pod "network-metrics-daemon-6cldf" (UID: "f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:33.123248 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:33.123239 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn podName:832da8b4-c715-4416-9881-77c2baa30314 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:34.123219108 +0000 UTC m=+4.010510407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5kn" (UniqueName: "kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn") pod "network-check-target-8gqbc" (UID: "832da8b4-c715-4416-9881-77c2baa30314") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:33.224211 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:33.224164 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:33.224419 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:33.224388 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:33.224497 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:33.224462 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret podName:e3390d32-503c-4e8f-ab8d-c1b3881f2df9 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:34.224443666 +0000 UTC m=+4.111734970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret") pod "global-pull-secret-syncer-rf7b9" (UID: "e3390d32-503c-4e8f-ab8d-c1b3881f2df9") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:33.563605 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:33.563554 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 16:49:33 +0000 UTC" deadline="2028-03-11 14:54:39.711289411 +0000 UTC" Jul 10 16:54:33.563605 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:33.563597 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14638h0m6.147697441s" Jul 10 16:54:33.847120 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:33.847090 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441cb5b6_d258_4ccc_a950_62b1179f6c1a.slice/crio-99204cc5f1f179a28f776bacd971047ed4762c43141458a6f5745fbbae032b14 WatchSource:0}: Error finding container 99204cc5f1f179a28f776bacd971047ed4762c43141458a6f5745fbbae032b14: Status 404 returned error can't find the container with id 99204cc5f1f179a28f776bacd971047ed4762c43141458a6f5745fbbae032b14 Jul 10 16:54:33.854854 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:33.854752 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36b025e_33b6_4b94_8b11_743ee8c1ebec.slice/crio-6951b976a04288543877c5af88ee48daa93ffa14bae1e97a2777619cfc404550 WatchSource:0}: Error finding container 6951b976a04288543877c5af88ee48daa93ffa14bae1e97a2777619cfc404550: Status 404 returned error can't find the container with id 6951b976a04288543877c5af88ee48daa93ffa14bae1e97a2777619cfc404550 Jul 10 16:54:33.855673 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:33.855339 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403bed03_bca0_461d_9cbf_767071e7a527.slice/crio-c2cc9acaad812457a5866e99a2749a1c0cc2c4ce76978c6cad6045415e99b4e9 WatchSource:0}: Error finding container c2cc9acaad812457a5866e99a2749a1c0cc2c4ce76978c6cad6045415e99b4e9: Status 404 returned error can't find the container with id c2cc9acaad812457a5866e99a2749a1c0cc2c4ce76978c6cad6045415e99b4e9 Jul 10 16:54:33.856855 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:33.856719 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda59a0ce_b509_4f6e_8685_b25d58a3fc2d.slice/crio-cfdc9cd1928fa110afec3e91e79ea167c2fc0b5c40a6fa36a7780b05863ea85d WatchSource:0}: Error finding container cfdc9cd1928fa110afec3e91e79ea167c2fc0b5c40a6fa36a7780b05863ea85d: Status 404 returned error can't find the container with id cfdc9cd1928fa110afec3e91e79ea167c2fc0b5c40a6fa36a7780b05863ea85d Jul 10 16:54:33.857614 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:33.857592 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a37165_0d14_486d_8ff2_bdbdf1818049.slice/crio-6dcd38adf30bc9021ee0906a54e72fa89abb5bd3e5e00d7044942903ba560e7e WatchSource:0}: Error finding container 6dcd38adf30bc9021ee0906a54e72fa89abb5bd3e5e00d7044942903ba560e7e: Status 404 returned error can't find the container with id 6dcd38adf30bc9021ee0906a54e72fa89abb5bd3e5e00d7044942903ba560e7e Jul 10 16:54:33.859760 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:54:33.859666 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac53225_b3f6_4a01_a823_03c876fdc239.slice/crio-b16115e1386ccaf59590870fca8ccba71457aa3ad13101fb90c6d80c7e28af43 WatchSource:0}: Error finding container b16115e1386ccaf59590870fca8ccba71457aa3ad13101fb90c6d80c7e28af43: Status 404 returned error can't find the container with id b16115e1386ccaf59590870fca8ccba71457aa3ad13101fb90c6d80c7e28af43 Jul 10 16:54:34.134565 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.134350 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5kn\" (UniqueName: \"kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn\") pod \"network-check-target-8gqbc\" (UID: \"832da8b4-c715-4416-9881-77c2baa30314\") " pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:34.134716 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.134624 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs\") pod \"network-metrics-daemon-6cldf\" (UID: \"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c\") " pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:34.134716 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:34.134545 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:54:34.134716 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:34.134659 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:54:34.134716 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:34.134672 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l5kn for pod openshift-network-diagnostics/network-check-target-8gqbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:34.134883 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:34.134727 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn podName:832da8b4-c715-4416-9881-77c2baa30314 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:36.134709427 +0000 UTC m=+6.022000745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5kn" (UniqueName: "kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn") pod "network-check-target-8gqbc" (UID: "832da8b4-c715-4416-9881-77c2baa30314") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:34.134883 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:34.134737 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:34.134883 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:34.134791 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs podName:f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c nodeName:}" failed. No retries permitted until 2026-07-10 16:54:36.134768266 +0000 UTC m=+6.022059565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs") pod "network-metrics-daemon-6cldf" (UID: "f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:34.200591 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.200540 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:54:34.235889 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.235853 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:34.236105 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:34.236005 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:34.236105 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:34.236078 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret podName:e3390d32-503c-4e8f-ab8d-c1b3881f2df9 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:36.236059859 +0000 UTC m=+6.123351165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret") pod "global-pull-secret-syncer-rf7b9" (UID: "e3390d32-503c-4e8f-ab8d-c1b3881f2df9") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:34.564855 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.564780 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 16:49:33 +0000 UTC" deadline="2028-04-21 18:59:13.202540468 +0000 UTC" Jul 10 16:54:34.564855 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.564852 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15626h4m38.637694114s" Jul 10 16:54:34.677021 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.676504 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:34.677021 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:34.676646 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:34.677021 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.676814 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:34.677021 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:34.676926 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:34.677650 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.677494 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:34.677650 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:34.677610 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:34.706599 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.706092 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-207.ec2.internal" event={"ID":"b203731d72488e0ccf24b1f1be91c047","Type":"ContainerStarted","Data":"dcf8b03a5187b35a0ae98b2b3a62c1391247bff9c5a27f9d97764d8ae2407c3a"} Jul 10 16:54:34.717606 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.717484 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" event={"ID":"da59a0ce-b509-4f6e-8685-b25d58a3fc2d","Type":"ContainerStarted","Data":"cfdc9cd1928fa110afec3e91e79ea167c2fc0b5c40a6fa36a7780b05863ea85d"} Jul 10 16:54:34.720966 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.720897 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-207.ec2.internal" podStartSLOduration=2.720878554 podStartE2EDuration="2.720878554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:34.720693177 +0000 UTC m=+4.607984501" watchObservedRunningTime="2026-07-10 16:54:34.720878554 +0000 UTC m=+4.608169871" Jul 10 16:54:34.727650 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.727555 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d62w2" event={"ID":"403bed03-bca0-461d-9cbf-767071e7a527","Type":"ContainerStarted","Data":"c2cc9acaad812457a5866e99a2749a1c0cc2c4ce76978c6cad6045415e99b4e9"} Jul 10 16:54:34.729539 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.729456 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kzbjz" event={"ID":"dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff","Type":"ContainerStarted","Data":"d42cd05cb7d6ca7cfd8f10509bfcf393cae44f89eb6df2c2e6fb3a228079517b"} Jul 10 16:54:34.750303 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.750212 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" event={"ID":"e54f7489-3734-4c1e-909f-bcee0f6f8ced","Type":"ContainerStarted","Data":"1b0d716d53b5dad7988354169f37d92c389224dd1397d1617ef1d267432774c8"} Jul 10 16:54:34.752714 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.752679 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n8qpc" event={"ID":"441cb5b6-d258-4ccc-a950-62b1179f6c1a","Type":"ContainerStarted","Data":"99204cc5f1f179a28f776bacd971047ed4762c43141458a6f5745fbbae032b14"} Jul 10 16:54:34.766482 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.765814 2567 generic.go:356] "Generic (PLEG): container finished" podID="27c9d0fc09f37a4d51b4693af721e488" containerID="9c9ad2cecc832d9800d2e06b46786ec2f2e7b639b6d93f5b612580d4ea029804" exitCode=0 Jul 10 16:54:34.766482 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.765907 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal" event={"ID":"27c9d0fc09f37a4d51b4693af721e488","Type":"ContainerDied","Data":"9c9ad2cecc832d9800d2e06b46786ec2f2e7b639b6d93f5b612580d4ea029804"} Jul 10 16:54:34.780967 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.780937 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g924d" event={"ID":"5ac53225-b3f6-4a01-a823-03c876fdc239","Type":"ContainerStarted","Data":"b16115e1386ccaf59590870fca8ccba71457aa3ad13101fb90c6d80c7e28af43"} Jul 10 16:54:34.792076 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.792043 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5jgps" event={"ID":"b8a37165-0d14-486d-8ff2-bdbdf1818049","Type":"ContainerStarted","Data":"6dcd38adf30bc9021ee0906a54e72fa89abb5bd3e5e00d7044942903ba560e7e"} Jul 10 16:54:34.800374 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.800343 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rnvf6" event={"ID":"f36b025e-33b6-4b94-8b11-743ee8c1ebec","Type":"ContainerStarted","Data":"6951b976a04288543877c5af88ee48daa93ffa14bae1e97a2777619cfc404550"} Jul 10 16:54:34.808281 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:34.808248 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4blh9" event={"ID":"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc","Type":"ContainerStarted","Data":"16fb310a04979cb107fce55cd6c460c1bf68ecbfbc2ad56b9c42f4fbb45fce11"} Jul 10 16:54:35.832499 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:35.832395 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal" event={"ID":"27c9d0fc09f37a4d51b4693af721e488","Type":"ContainerStarted","Data":"16b2861a39818117b649a3d856e11a834f631200d4b4c022d692dd5408b78516"} Jul 10 16:54:35.849015 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:35.848719 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-207.ec2.internal" podStartSLOduration=3.848694499 podStartE2EDuration="3.848694499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:35.847621844 +0000 UTC m=+5.734913169" watchObservedRunningTime="2026-07-10 16:54:35.848694499 +0000 UTC m=+5.735985824" Jul 10 16:54:36.152816 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:36.152725 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs\") pod \"network-metrics-daemon-6cldf\" (UID: \"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c\") " pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:36.152816 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:36.152789 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5kn\" (UniqueName: \"kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn\") pod \"network-check-target-8gqbc\" (UID: \"832da8b4-c715-4416-9881-77c2baa30314\") " pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:36.153047 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:36.152913 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:36.153047 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:36.152972 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:54:36.153047 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:36.152996 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:54:36.153047 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:36.153013 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l5kn for pod openshift-network-diagnostics/network-check-target-8gqbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:36.153047 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:36.152996 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs podName:f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c nodeName:}" failed. No retries permitted until 2026-07-10 16:54:40.152973237 +0000 UTC m=+10.040264540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs") pod "network-metrics-daemon-6cldf" (UID: "f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:36.153323 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:36.153076 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn podName:832da8b4-c715-4416-9881-77c2baa30314 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:40.153057964 +0000 UTC m=+10.040349271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5kn" (UniqueName: "kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn") pod "network-check-target-8gqbc" (UID: "832da8b4-c715-4416-9881-77c2baa30314") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:36.253752 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:36.253706 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:36.253935 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:36.253922 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:36.254016 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:36.253990 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret podName:e3390d32-503c-4e8f-ab8d-c1b3881f2df9 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:40.253968943 +0000 UTC m=+10.141260248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret") pod "global-pull-secret-syncer-rf7b9" (UID: "e3390d32-503c-4e8f-ab8d-c1b3881f2df9") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:36.680560 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:36.680526 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:36.686322 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:36.680811 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:36.686322 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:36.680862 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:36.686565 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:36.686458 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:36.686996 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:36.686798 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:36.686996 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:36.686943 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:38.676495 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:38.675870 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:38.676495 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:38.676018 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:38.676495 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:38.676429 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:38.677113 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:38.676538 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:38.677113 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:38.676571 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:38.677113 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:38.676713 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:40.187091 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:40.187050 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs\") pod \"network-metrics-daemon-6cldf\" (UID: \"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c\") " pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:40.187091 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:40.187097 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5kn\" (UniqueName: \"kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn\") pod \"network-check-target-8gqbc\" (UID: \"832da8b4-c715-4416-9881-77c2baa30314\") " pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:40.187595 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:40.187226 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:40.187595 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:40.187238 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:54:40.187595 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:40.187254 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:54:40.187595 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:40.187267 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l5kn for pod openshift-network-diagnostics/network-check-target-8gqbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:40.187595 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:40.187319 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs podName:f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c nodeName:}" failed. No retries permitted until 2026-07-10 16:54:48.187297141 +0000 UTC m=+18.074588456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs") pod "network-metrics-daemon-6cldf" (UID: "f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:40.187595 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:40.187339 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn podName:832da8b4-c715-4416-9881-77c2baa30314 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:48.187328465 +0000 UTC m=+18.074619768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5kn" (UniqueName: "kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn") pod "network-check-target-8gqbc" (UID: "832da8b4-c715-4416-9881-77c2baa30314") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:40.288233 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:40.288189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:40.288414 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:40.288381 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:40.288476 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:40.288445 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret podName:e3390d32-503c-4e8f-ab8d-c1b3881f2df9 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:48.288431374 +0000 UTC m=+18.175722673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret") pod "global-pull-secret-syncer-rf7b9" (UID: "e3390d32-503c-4e8f-ab8d-c1b3881f2df9") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:40.677488 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:40.677358 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:40.677488 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:40.677358 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:40.679200 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:40.678594 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:40.679200 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:40.678708 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:40.679200 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:40.679068 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:40.679200 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:40.679154 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:42.676447 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:42.676400 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:42.676996 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:42.676398 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:42.676996 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:42.676551 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:42.676996 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:42.676647 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:42.676996 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:42.676713 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:42.676996 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:42.676778 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:44.676464 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:44.676434 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:44.676924 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:44.676470 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:44.676924 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:44.676440 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:44.676924 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:44.676565 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:44.676924 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:44.676623 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:44.676924 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:44.676712 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:46.676337 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:46.676302 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:46.676753 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:46.676302 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:46.676753 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:46.676431 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:46.676753 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:46.676457 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:46.676753 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:46.676553 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:46.676753 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:46.676641 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:48.252576 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:48.252537 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs\") pod \"network-metrics-daemon-6cldf\" (UID: \"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c\") " pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:48.253007 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:48.252589 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5kn\" (UniqueName: \"kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn\") pod \"network-check-target-8gqbc\" (UID: \"832da8b4-c715-4416-9881-77c2baa30314\") " pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:48.253007 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:48.252715 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:48.253007 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:48.252797 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs podName:f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c nodeName:}" failed. No retries permitted until 2026-07-10 16:55:04.25277671 +0000 UTC m=+34.140068014 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs") pod "network-metrics-daemon-6cldf" (UID: "f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:48.253007 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:48.252725 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:54:48.253007 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:48.252857 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:54:48.253007 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:48.252875 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l5kn for pod openshift-network-diagnostics/network-check-target-8gqbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:48.253007 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:48.252935 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn podName:832da8b4-c715-4416-9881-77c2baa30314 nodeName:}" failed. No retries permitted until 2026-07-10 16:55:04.252918587 +0000 UTC m=+34.140209905 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5kn" (UniqueName: "kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn") pod "network-check-target-8gqbc" (UID: "832da8b4-c715-4416-9881-77c2baa30314") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:48.353082 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:48.353038 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:48.353296 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:48.353234 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:48.353355 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:48.353316 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret podName:e3390d32-503c-4e8f-ab8d-c1b3881f2df9 nodeName:}" failed. No retries permitted until 2026-07-10 16:55:04.353298553 +0000 UTC m=+34.240589854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret") pod "global-pull-secret-syncer-rf7b9" (UID: "e3390d32-503c-4e8f-ab8d-c1b3881f2df9") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:48.676142 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:48.676046 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:48.676142 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:48.676083 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:48.676142 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:48.676184 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:48.676744 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:48.676246 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:48.676744 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:48.676397 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:48.676744 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:48.676519 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:50.676472 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:50.676436 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:50.676908 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:50.676553 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:50.676908 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:50.676558 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:50.676908 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:50.676670 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:50.676908 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:50.676706 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:50.676908 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:50.676795 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:52.676367 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:52.676326 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:52.676954 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:52.676326 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:52.676954 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:52.676476 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:52.676954 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:52.676338 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:52.676954 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:52.676545 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:52.676954 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:52.676694 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:53.875184 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.874933 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" event={"ID":"da59a0ce-b509-4f6e-8685-b25d58a3fc2d","Type":"ContainerStarted","Data":"adce5b54ec81fb57af276c6c1f69675742ec51ec4186df7526632889af378570"} Jul 10 16:54:53.879756 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.876846 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d62w2" event={"ID":"403bed03-bca0-461d-9cbf-767071e7a527","Type":"ContainerStarted","Data":"5a27356a37be9022f996b8c5b124636758465847893d0e52b9c844534ec379fc"} Jul 10 16:54:53.879756 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.878299 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kzbjz" event={"ID":"dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff","Type":"ContainerStarted","Data":"eefb1e07bd2c6056ffba31977d0fe682243d63b281a46ad2166593e360d46cc6"} Jul 10 16:54:53.879756 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.879721 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" event={"ID":"e54f7489-3734-4c1e-909f-bcee0f6f8ced","Type":"ContainerStarted","Data":"5de8336854da2036a0d00a00a9a0ec8c97f6f7de81c4a38878013d10ae4e755f"} Jul 10 16:54:53.881228 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.881197 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n8qpc" event={"ID":"441cb5b6-d258-4ccc-a950-62b1179f6c1a","Type":"ContainerStarted","Data":"57fc62baa2f624e81770161c5ef67fb5bb8bf5441556d1ab122af2937943d747"} Jul 10 16:54:53.882736 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.882702 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g924d" event={"ID":"5ac53225-b3f6-4a01-a823-03c876fdc239","Type":"ContainerStarted","Data":"95e2e448efde9b91050c94b1aa0b62897e38df4e768acf5d9ea38e7371926a83"} Jul 10 16:54:53.884511 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.884485 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5jgps" event={"ID":"b8a37165-0d14-486d-8ff2-bdbdf1818049","Type":"ContainerStarted","Data":"36843acc7bd260b17560ea43b8fae7cf557da66d9fbe6c38f0ac8b689166c06b"} Jul 10 16:54:53.886318 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.886289 2567 generic.go:356] "Generic (PLEG): container finished" podID="b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc" containerID="ce0e6a1dd0738d1d9948845dc07c324d77df869ed91fb1b8886ed830154f8e7e" exitCode=0 Jul 10 16:54:53.886421 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.886338 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4blh9" event={"ID":"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc","Type":"ContainerDied","Data":"ce0e6a1dd0738d1d9948845dc07c324d77df869ed91fb1b8886ed830154f8e7e"} Jul 10 16:54:53.902345 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.902300 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d62w2" podStartSLOduration=3.247424282 podStartE2EDuration="22.902261662s" totalImagesPullingTime="19.65483738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:53.901749348 +0000 UTC m=+23.789040669" watchObservedRunningTime="2026-07-10 16:54:53.902261662 +0000 UTC m=+23.789552981" Jul 10 16:54:53.916884 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.916842 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kzbjz" podStartSLOduration=3.2711503459999998 podStartE2EDuration="22.916828496s" totalImagesPullingTime="19.64567815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:53.916407543 +0000 UTC m=+23.803698888" watchObservedRunningTime="2026-07-10 16:54:53.916828496 +0000 UTC m=+23.804119817" Jul 10 16:54:53.930366 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.930316 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n8qpc" podStartSLOduration=3.326361184 podStartE2EDuration="22.93026401s" totalImagesPullingTime="19.603902826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:53.929562715 +0000 UTC m=+23.816854048" watchObservedRunningTime="2026-07-10 16:54:53.93026401 +0000 UTC m=+23.817555330" Jul 10 16:54:53.946367 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.946315 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5jgps" podStartSLOduration=3.304105616 podStartE2EDuration="22.946258255s" totalImagesPullingTime="19.642152639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:53.945698719 +0000 UTC m=+23.832990039" watchObservedRunningTime="2026-07-10 16:54:53.946258255 +0000 UTC m=+23.833549575" Jul 10 16:54:53.988104 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:53.988064 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g924d" podStartSLOduration=8.121309007 podStartE2EDuration="22.988048207s" totalImagesPullingTime="14.8667392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:53.987513033 +0000 UTC m=+23.874804371" watchObservedRunningTime="2026-07-10 16:54:53.988048207 +0000 UTC m=+23.875339527" Jul 10 16:54:54.676656 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:54.676371 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:54.676807 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:54.676370 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:54.676807 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:54.676734 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:54.676807 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:54.676422 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:54.676936 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:54.676805 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:54.676936 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:54.676870 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:54.866904 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:54.866878 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 16:54:54.891765 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:54.891741 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 16:54:54.892213 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:54.892192 2567 generic.go:356] "Generic (PLEG): container finished" podID="da59a0ce-b509-4f6e-8685-b25d58a3fc2d" containerID="b0187f9027723d9f4e0b083479addca41d968e55c97e6127ab081b6cb569128d" exitCode=1 Jul 10 16:54:54.892313 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:54.892286 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" event={"ID":"da59a0ce-b509-4f6e-8685-b25d58a3fc2d","Type":"ContainerStarted","Data":"9462bfe0e3446da3207d86bb8edba634de56cd27e6b115878ccfe47ee0a5f14a"} Jul 10 16:54:54.892401 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:54.892384 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" event={"ID":"da59a0ce-b509-4f6e-8685-b25d58a3fc2d","Type":"ContainerStarted","Data":"7612b1dd4644b7cd16fe4fc22db16c890931d1f14642a46fcc608ecc56544622"} Jul 10 16:54:54.892458 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:54.892433 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" event={"ID":"da59a0ce-b509-4f6e-8685-b25d58a3fc2d","Type":"ContainerStarted","Data":"d57eec117cdf9cd67dfe83a4e25109cbce64151cc357498154e63981a1ed68c7"} Jul 10 16:54:54.892499 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:54.892480 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" event={"ID":"da59a0ce-b509-4f6e-8685-b25d58a3fc2d","Type":"ContainerStarted","Data":"7b5f068cad04a6ea919fdf3abb9b93e274dbd9d80b73c4954ef10133f3c4dc88"} Jul 10 16:54:54.892539 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:54.892526 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" event={"ID":"da59a0ce-b509-4f6e-8685-b25d58a3fc2d","Type":"ContainerDied","Data":"b0187f9027723d9f4e0b083479addca41d968e55c97e6127ab081b6cb569128d"} Jul 10 16:54:54.894337 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:54.894285 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" event={"ID":"e54f7489-3734-4c1e-909f-bcee0f6f8ced","Type":"ContainerStarted","Data":"d3ad0cef295fddc071f9e9c05c8ce138d7150701cc08df62dd9a833261f3d572"} Jul 10 16:54:55.566706 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:55.566599 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T16:54:54.86689939Z","UUID":"20859b17-7f1f-4cb0-a61e-af0c9dbaaf1c","Handler":null,"Name":"","Endpoint":""} Jul 10 16:54:55.568671 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:55.568648 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 16:54:55.568796 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:55.568698 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 16:54:55.899253 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:55.899219 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rnvf6" event={"ID":"f36b025e-33b6-4b94-8b11-743ee8c1ebec","Type":"ContainerStarted","Data":"f53c184a65c22cdc5997f8088042aba7e56be3ef5e2ef01b4e75be93c33e8b8c"} Jul 10 16:54:55.925797 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:55.925750 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rnvf6" podStartSLOduration=5.283168145 podStartE2EDuration="24.925734197s" totalImagesPullingTime="19.642566052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:55.925566278 +0000 UTC m=+25.812857595" watchObservedRunningTime="2026-07-10 16:54:55.925734197 +0000 UTC m=+25.813025518" Jul 10 16:54:56.676415 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:56.676198 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:56.676575 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:56.676236 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:56.676575 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:56.676515 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:56.676700 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:56.676283 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:56.676700 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:56.676602 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:56.676700 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:56.676673 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:56.905943 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:56.905915 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 16:54:56.906674 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:56.906625 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" event={"ID":"da59a0ce-b509-4f6e-8685-b25d58a3fc2d","Type":"ContainerStarted","Data":"30158afb144bf1cc31f4a390b8add4f1b688a6c6282928ba138ee9208934fbaf"} Jul 10 16:54:56.909395 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:56.909367 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" event={"ID":"e54f7489-3734-4c1e-909f-bcee0f6f8ced","Type":"ContainerStarted","Data":"9042032f5e10b4cd19dffdfb10005b9d275ea05f0c21be1b52031575fd489ac7"} Jul 10 16:54:56.933424 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:56.933338 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxnqg" podStartSLOduration=4.330892306 podStartE2EDuration="25.933322031s" totalImagesPullingTime="21.602429725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:56.932664101 +0000 UTC m=+26.819955420" watchObservedRunningTime="2026-07-10 16:54:56.933322031 +0000 UTC m=+26.820613351" Jul 10 16:54:58.593479 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.593225 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n8qpc" Jul 10 16:54:58.594128 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.593943 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n8qpc" Jul 10 16:54:58.675546 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.675508 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:54:58.675709 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.675509 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:54:58.675709 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:58.675641 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:54:58.675709 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.675523 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:54:58.675861 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:58.675721 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:54:58.675861 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:54:58.675781 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:54:58.916727 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.916626 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 16:54:58.917037 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.917013 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" event={"ID":"da59a0ce-b509-4f6e-8685-b25d58a3fc2d","Type":"ContainerStarted","Data":"f81acda78cffffc75ed1614351f5ff26e38e9cdfdee4761967902a6db1ecd4b3"} Jul 10 16:54:58.917420 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.917386 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:58.917519 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.917471 2567 scope.go:117] "RemoveContainer" containerID="b0187f9027723d9f4e0b083479addca41d968e55c97e6127ab081b6cb569128d" Jul 10 16:54:58.917600 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.917476 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:58.917650 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.917619 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:58.918983 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.918958 2567 generic.go:356] "Generic (PLEG): container finished" podID="b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc" containerID="285dd9f0dc0f3d21537adeb72bcf7bbd2009d7eacb28cdda75a70299db035057" exitCode=0 Jul 10 16:54:58.919088 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.919049 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4blh9" event={"ID":"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc","Type":"ContainerDied","Data":"285dd9f0dc0f3d21537adeb72bcf7bbd2009d7eacb28cdda75a70299db035057"} Jul 10 16:54:58.919313 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.919259 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n8qpc" Jul 10 16:54:58.919883 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.919854 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n8qpc" Jul 10 16:54:58.934314 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.934232 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:58.934624 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:58.934601 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:54:59.924000 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:59.923978 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 16:54:59.924379 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:59.924301 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" event={"ID":"da59a0ce-b509-4f6e-8685-b25d58a3fc2d","Type":"ContainerStarted","Data":"d9bb8a5c77a19df95e3b9e842d590c0c6abeb06e3ad9696d7c933f9abe2fb2bc"} Jul 10 16:54:59.953905 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:54:59.953855 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" podStartSLOduration=9.262635922 podStartE2EDuration="28.953839856s" totalImagesPullingTime="19.691203934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:59.95263205 +0000 UTC m=+29.839923403" watchObservedRunningTime="2026-07-10 16:54:59.953839856 +0000 UTC m=+29.841131176" Jul 10 16:55:00.417825 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:00.417797 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rf7b9"] Jul 10 16:55:00.417978 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:00.417901 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:55:00.418027 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:00.417997 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:55:00.420617 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:00.420591 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6cldf"] Jul 10 16:55:00.420742 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:00.420698 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:55:00.420820 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:00.420798 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:55:00.421206 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:00.421183 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8gqbc"] Jul 10 16:55:00.421325 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:00.421296 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:55:00.421403 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:00.421386 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:55:00.927869 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:00.927838 2567 generic.go:356] "Generic (PLEG): container finished" podID="b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc" containerID="0854954718e3dfd6b261ae2fb5a8bf0516626741047472004ede07e33c681c89" exitCode=0 Jul 10 16:55:00.928265 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:00.927928 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4blh9" event={"ID":"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc","Type":"ContainerDied","Data":"0854954718e3dfd6b261ae2fb5a8bf0516626741047472004ede07e33c681c89"} Jul 10 16:55:01.676085 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:01.675886 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:55:01.676295 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:01.675887 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:55:01.676295 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:01.676191 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:55:01.676295 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:01.676264 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:55:02.676053 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:02.675973 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:55:02.676518 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:02.676112 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:55:02.935307 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:02.935211 2567 generic.go:356] "Generic (PLEG): container finished" podID="b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc" containerID="7b5489be89d5234b4b8f4282a22f99223bafcc1d596d286a1568b95c8898b3a2" exitCode=0 Jul 10 16:55:02.935307 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:02.935303 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4blh9" event={"ID":"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc","Type":"ContainerDied","Data":"7b5489be89d5234b4b8f4282a22f99223bafcc1d596d286a1568b95c8898b3a2"} Jul 10 16:55:03.675537 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:03.675491 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:55:03.675714 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:03.675627 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:55:03.675714 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:03.675693 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:55:03.675819 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:03.675797 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:55:04.273206 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:04.273169 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs\") pod \"network-metrics-daemon-6cldf\" (UID: \"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c\") " pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:55:04.273781 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:04.273218 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5kn\" (UniqueName: \"kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn\") pod \"network-check-target-8gqbc\" (UID: \"832da8b4-c715-4416-9881-77c2baa30314\") " pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:55:04.273781 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:04.273332 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:55:04.273781 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:04.273351 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:55:04.273781 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:04.273369 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:55:04.273781 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:04.273381 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l5kn for pod openshift-network-diagnostics/network-check-target-8gqbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:55:04.273781 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:04.273404 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs podName:f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c nodeName:}" failed. No retries permitted until 2026-07-10 16:55:36.27338815 +0000 UTC m=+66.160679452 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs") pod "network-metrics-daemon-6cldf" (UID: "f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:55:04.273781 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:04.273425 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn podName:832da8b4-c715-4416-9881-77c2baa30314 nodeName:}" failed. No retries permitted until 2026-07-10 16:55:36.273412708 +0000 UTC m=+66.160704008 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5kn" (UniqueName: "kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn") pod "network-check-target-8gqbc" (UID: "832da8b4-c715-4416-9881-77c2baa30314") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:55:04.374116 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:04.374082 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:55:04.374311 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:04.374243 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:55:04.374368 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:04.374338 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret podName:e3390d32-503c-4e8f-ab8d-c1b3881f2df9 nodeName:}" failed. No retries permitted until 2026-07-10 16:55:36.374318022 +0000 UTC m=+66.261609324 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret") pod "global-pull-secret-syncer-rf7b9" (UID: "e3390d32-503c-4e8f-ab8d-c1b3881f2df9") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:55:04.676411 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:04.676320 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:55:04.676585 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:04.676452 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqbc" podUID="832da8b4-c715-4416-9881-77c2baa30314" Jul 10 16:55:05.300806 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:05.300776 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kzbjz_dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff/dns-node-resolver/0.log" Jul 10 16:55:05.676206 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:05.676122 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:55:05.676206 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:05.676144 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:55:05.676445 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:05.676263 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cldf" podUID="f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c" Jul 10 16:55:05.676445 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:05.676369 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf7b9" podUID="e3390d32-503c-4e8f-ab8d-c1b3881f2df9" Jul 10 16:55:06.275050 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:06.275025 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g924d_5ac53225-b3f6-4a01-a823-03c876fdc239/node-ca/0.log" Jul 10 16:55:06.406859 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:06.406787 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-207.ec2.internal" event="NodeReady" Jul 10 16:55:06.407315 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:06.406913 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 16:55:06.678796 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:06.678722 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:55:06.683526 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:06.683506 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 16:55:06.683609 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:06.683508 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 16:55:06.683949 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:06.683934 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jn8d4\"" Jul 10 16:55:07.676347 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:07.676308 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:55:07.676347 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:07.676355 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:55:07.679022 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:07.678987 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 16:55:07.679022 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:07.679007 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdmq6\"" Jul 10 16:55:07.679196 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:07.679037 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 16:55:09.952457 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:09.952243 2567 generic.go:356] "Generic (PLEG): container finished" podID="b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc" containerID="285e6086d2265c75720fdbf104c6aafb1f40bd3695f9c14fa3b2c8dc1f98ec4c" exitCode=0 Jul 10 16:55:09.952781 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:09.952305 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4blh9" event={"ID":"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc","Type":"ContainerDied","Data":"285e6086d2265c75720fdbf104c6aafb1f40bd3695f9c14fa3b2c8dc1f98ec4c"} Jul 10 16:55:10.958303 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:10.958252 2567 generic.go:356] "Generic (PLEG): container finished" podID="b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc" containerID="47f20a587372b1e8112df9914be312216bc5a7b247f4b79d967f51824801c0e7" exitCode=0 Jul 10 16:55:10.958811 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:10.958324 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4blh9" event={"ID":"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc","Type":"ContainerDied","Data":"47f20a587372b1e8112df9914be312216bc5a7b247f4b79d967f51824801c0e7"} Jul 10 16:55:11.966778 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:11.966740 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4blh9" event={"ID":"b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc","Type":"ContainerStarted","Data":"8a37bfc57b5f966761104d5ec1e9a18fca585d3d4ff6b8526d0875fc0fb6e134"} Jul 10 16:55:11.994806 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:11.994748 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4blh9" podStartSLOduration=8.02187869 podStartE2EDuration="40.994728185s" totalImagesPullingTime="32.972849495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:11.994258763 +0000 UTC m=+41.881550127" watchObservedRunningTime="2026-07-10 16:55:11.994728185 +0000 UTC m=+41.882019582" Jul 10 16:55:29.668405 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.668362 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-j4xzk"] Jul 10 16:55:29.713606 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.713566 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5"] Jul 10 16:55:29.713748 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.713685 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-j4xzk" Jul 10 16:55:29.715929 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.715904 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 10 16:55:29.716673 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.716652 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 10 16:55:29.716770 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.716752 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tlcjl\"" Jul 10 16:55:29.727053 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.727034 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b"] Jul 10 16:55:29.727147 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.727133 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.731096 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.731077 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 16:55:29.731207 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.731172 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 16:55:29.731207 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.731191 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 16:55:29.731207 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.731205 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 16:55:29.731444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.731420 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 16:55:29.731580 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.731447 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 16:55:29.731580 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.731431 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 16:55:29.741052 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.741034 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-j4xzk"] Jul 10 16:55:29.741133 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.741064 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b"] Jul 10 16:55:29.741133 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.741079 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5"] Jul 10 16:55:29.741198 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.741150 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" Jul 10 16:55:29.744958 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.744938 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 10 16:55:29.753980 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.753960 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f032d76f-2506-46c0-ab02-2bfa01906b9a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.754070 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.753989 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/473995a2-4644-48b2-bb05-cc5aadfcc990-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-j4xzk\" (UID: \"473995a2-4644-48b2-bb05-cc5aadfcc990\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-j4xzk" Jul 10 16:55:29.754070 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.754021 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tnbf\" (UniqueName: \"kubernetes.io/projected/f032d76f-2506-46c0-ab02-2bfa01906b9a-kube-api-access-4tnbf\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.754070 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.754065 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/473995a2-4644-48b2-bb05-cc5aadfcc990-nginx-conf\") pod \"networking-console-plugin-6cf856758c-j4xzk\" (UID: \"473995a2-4644-48b2-bb05-cc5aadfcc990\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-j4xzk" Jul 10 16:55:29.754187 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.754123 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b67970-933f-4673-b9f3-1dcdf445dcf1-tmp\") pod \"klusterlet-addon-workmgr-cb47698f9-fd57b\" (UID: \"e0b67970-933f-4673-b9f3-1dcdf445dcf1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" Jul 10 16:55:29.754187 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.754138 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f032d76f-2506-46c0-ab02-2bfa01906b9a-hub\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.754187 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.754152 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e0b67970-933f-4673-b9f3-1dcdf445dcf1-klusterlet-config\") pod \"klusterlet-addon-workmgr-cb47698f9-fd57b\" (UID: \"e0b67970-933f-4673-b9f3-1dcdf445dcf1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" Jul 10 16:55:29.754187 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.754165 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f032d76f-2506-46c0-ab02-2bfa01906b9a-ca\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.754187 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.754179 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f032d76f-2506-46c0-ab02-2bfa01906b9a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.754418 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.754202 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f032d76f-2506-46c0-ab02-2bfa01906b9a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.754418 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.754282 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbz9s\" (UniqueName: \"kubernetes.io/projected/e0b67970-933f-4673-b9f3-1dcdf445dcf1-kube-api-access-fbz9s\") pod \"klusterlet-addon-workmgr-cb47698f9-fd57b\" (UID: \"e0b67970-933f-4673-b9f3-1dcdf445dcf1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" Jul 10 16:55:29.766499 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.766471 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c64cd"] Jul 10 16:55:29.786109 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.786076 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m5jwp"] Jul 10 16:55:29.786291 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.786200 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c64cd" Jul 10 16:55:29.788685 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.788663 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 16:55:29.789381 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.789360 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 16:55:29.789471 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.789423 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b5v82\"" Jul 10 16:55:29.789471 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.789428 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 16:55:29.801149 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.801128 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m5jwp"] Jul 10 16:55:29.801253 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.801161 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c64cd"] Jul 10 16:55:29.801345 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.801291 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:29.804022 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.804005 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ccqfl\"" Jul 10 16:55:29.804156 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.804135 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 16:55:29.804235 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.804222 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 16:55:29.854880 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.854843 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnbf\" (UniqueName: \"kubernetes.io/projected/f032d76f-2506-46c0-ab02-2bfa01906b9a-kube-api-access-4tnbf\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.855066 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.854888 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1c389de-199e-468c-851d-1a5d8a9e51fe-metrics-tls\") pod \"dns-default-m5jwp\" (UID: \"f1c389de-199e-468c-851d-1a5d8a9e51fe\") " pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:29.855066 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.854908 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c389de-199e-468c-851d-1a5d8a9e51fe-config-volume\") pod \"dns-default-m5jwp\" (UID: \"f1c389de-199e-468c-851d-1a5d8a9e51fe\") " pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:29.855066 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.854924 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmpcz\" (UniqueName: \"kubernetes.io/projected/f1c389de-199e-468c-851d-1a5d8a9e51fe-kube-api-access-rmpcz\") pod \"dns-default-m5jwp\" (UID: \"f1c389de-199e-468c-851d-1a5d8a9e51fe\") " pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:29.855066 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.854991 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/473995a2-4644-48b2-bb05-cc5aadfcc990-nginx-conf\") pod \"networking-console-plugin-6cf856758c-j4xzk\" (UID: \"473995a2-4644-48b2-bb05-cc5aadfcc990\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-j4xzk" Jul 10 16:55:29.855066 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855026 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c389de-199e-468c-851d-1a5d8a9e51fe-tmp-dir\") pod \"dns-default-m5jwp\" (UID: \"f1c389de-199e-468c-851d-1a5d8a9e51fe\") " pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:29.855362 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855071 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b67970-933f-4673-b9f3-1dcdf445dcf1-tmp\") pod \"klusterlet-addon-workmgr-cb47698f9-fd57b\" (UID: \"e0b67970-933f-4673-b9f3-1dcdf445dcf1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" Jul 10 16:55:29.855362 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855094 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f032d76f-2506-46c0-ab02-2bfa01906b9a-hub\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.855362 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855131 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e0b67970-933f-4673-b9f3-1dcdf445dcf1-klusterlet-config\") pod \"klusterlet-addon-workmgr-cb47698f9-fd57b\" (UID: \"e0b67970-933f-4673-b9f3-1dcdf445dcf1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" Jul 10 16:55:29.855362 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855147 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f032d76f-2506-46c0-ab02-2bfa01906b9a-ca\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.855362 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855162 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f032d76f-2506-46c0-ab02-2bfa01906b9a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.855362 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855180 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/853903d4-a52c-4bc3-a1ae-20a096602158-cert\") pod \"ingress-canary-c64cd\" (UID: \"853903d4-a52c-4bc3-a1ae-20a096602158\") " pod="openshift-ingress-canary/ingress-canary-c64cd" Jul 10 16:55:29.855362 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855209 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f032d76f-2506-46c0-ab02-2bfa01906b9a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.855362 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855227 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbz9s\" (UniqueName: \"kubernetes.io/projected/e0b67970-933f-4673-b9f3-1dcdf445dcf1-kube-api-access-fbz9s\") pod \"klusterlet-addon-workmgr-cb47698f9-fd57b\" (UID: \"e0b67970-933f-4673-b9f3-1dcdf445dcf1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" Jul 10 16:55:29.855362 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855248 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfpkd\" (UniqueName: \"kubernetes.io/projected/853903d4-a52c-4bc3-a1ae-20a096602158-kube-api-access-hfpkd\") pod \"ingress-canary-c64cd\" (UID: \"853903d4-a52c-4bc3-a1ae-20a096602158\") " pod="openshift-ingress-canary/ingress-canary-c64cd" Jul 10 16:55:29.855758 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855533 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b67970-933f-4673-b9f3-1dcdf445dcf1-tmp\") pod \"klusterlet-addon-workmgr-cb47698f9-fd57b\" (UID: \"e0b67970-933f-4673-b9f3-1dcdf445dcf1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" Jul 10 16:55:29.855758 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855581 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f032d76f-2506-46c0-ab02-2bfa01906b9a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.855758 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855621 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/473995a2-4644-48b2-bb05-cc5aadfcc990-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-j4xzk\" (UID: \"473995a2-4644-48b2-bb05-cc5aadfcc990\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-j4xzk" Jul 10 16:55:29.855993 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.855966 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/473995a2-4644-48b2-bb05-cc5aadfcc990-nginx-conf\") pod \"networking-console-plugin-6cf856758c-j4xzk\" (UID: \"473995a2-4644-48b2-bb05-cc5aadfcc990\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-j4xzk" Jul 10 16:55:29.856217 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.856191 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f032d76f-2506-46c0-ab02-2bfa01906b9a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.860482 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.860457 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/473995a2-4644-48b2-bb05-cc5aadfcc990-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-j4xzk\" (UID: \"473995a2-4644-48b2-bb05-cc5aadfcc990\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-j4xzk" Jul 10 16:55:29.860676 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.860658 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e0b67970-933f-4673-b9f3-1dcdf445dcf1-klusterlet-config\") pod \"klusterlet-addon-workmgr-cb47698f9-fd57b\" (UID: \"e0b67970-933f-4673-b9f3-1dcdf445dcf1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" Jul 10 16:55:29.860960 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.860941 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f032d76f-2506-46c0-ab02-2bfa01906b9a-ca\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.861039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.861019 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f032d76f-2506-46c0-ab02-2bfa01906b9a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.861123 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.861104 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f032d76f-2506-46c0-ab02-2bfa01906b9a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.861168 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.861104 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f032d76f-2506-46c0-ab02-2bfa01906b9a-hub\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.863845 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.863822 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n4jsc"] Jul 10 16:55:29.870415 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.870389 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbz9s\" (UniqueName: \"kubernetes.io/projected/e0b67970-933f-4673-b9f3-1dcdf445dcf1-kube-api-access-fbz9s\") pod \"klusterlet-addon-workmgr-cb47698f9-fd57b\" (UID: \"e0b67970-933f-4673-b9f3-1dcdf445dcf1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" Jul 10 16:55:29.872814 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.872788 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnbf\" (UniqueName: \"kubernetes.io/projected/f032d76f-2506-46c0-ab02-2bfa01906b9a-kube-api-access-4tnbf\") pod \"cluster-proxy-proxy-agent-54db746db9-lwpz5\" (UID: \"f032d76f-2506-46c0-ab02-2bfa01906b9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:29.893521 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.893493 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n4jsc"] Jul 10 16:55:29.893696 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.893642 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:29.896474 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.896453 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 16:55:29.896759 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.896744 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 16:55:29.896812 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.896749 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vsnsq\"" Jul 10 16:55:29.896978 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.896966 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 16:55:29.897298 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.897260 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 16:55:29.956554 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.956524 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/853903d4-a52c-4bc3-a1ae-20a096602158-cert\") pod \"ingress-canary-c64cd\" (UID: \"853903d4-a52c-4bc3-a1ae-20a096602158\") " pod="openshift-ingress-canary/ingress-canary-c64cd" Jul 10 16:55:29.956743 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.956559 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:29.956743 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.956587 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-crio-socket\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:29.956743 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.956608 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfpkd\" (UniqueName: \"kubernetes.io/projected/853903d4-a52c-4bc3-a1ae-20a096602158-kube-api-access-hfpkd\") pod \"ingress-canary-c64cd\" (UID: \"853903d4-a52c-4bc3-a1ae-20a096602158\") " pod="openshift-ingress-canary/ingress-canary-c64cd" Jul 10 16:55:29.956743 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.956697 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1c389de-199e-468c-851d-1a5d8a9e51fe-metrics-tls\") pod \"dns-default-m5jwp\" (UID: \"f1c389de-199e-468c-851d-1a5d8a9e51fe\") " pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:29.956743 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.956726 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:29.956993 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.956748 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c389de-199e-468c-851d-1a5d8a9e51fe-config-volume\") pod \"dns-default-m5jwp\" (UID: \"f1c389de-199e-468c-851d-1a5d8a9e51fe\") " pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:29.956993 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.956764 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmpcz\" (UniqueName: \"kubernetes.io/projected/f1c389de-199e-468c-851d-1a5d8a9e51fe-kube-api-access-rmpcz\") pod \"dns-default-m5jwp\" (UID: \"f1c389de-199e-468c-851d-1a5d8a9e51fe\") " pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:29.956993 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.956782 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-data-volume\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:29.956993 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.956910 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c389de-199e-468c-851d-1a5d8a9e51fe-tmp-dir\") pod \"dns-default-m5jwp\" (UID: \"f1c389de-199e-468c-851d-1a5d8a9e51fe\") " pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:29.956993 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.956969 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfr45\" (UniqueName: \"kubernetes.io/projected/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-kube-api-access-jfr45\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:29.957220 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.957194 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c389de-199e-468c-851d-1a5d8a9e51fe-tmp-dir\") pod \"dns-default-m5jwp\" (UID: \"f1c389de-199e-468c-851d-1a5d8a9e51fe\") " pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:29.957329 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.957312 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c389de-199e-468c-851d-1a5d8a9e51fe-config-volume\") pod \"dns-default-m5jwp\" (UID: \"f1c389de-199e-468c-851d-1a5d8a9e51fe\") " pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:29.959214 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.959184 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1c389de-199e-468c-851d-1a5d8a9e51fe-metrics-tls\") pod \"dns-default-m5jwp\" (UID: \"f1c389de-199e-468c-851d-1a5d8a9e51fe\") " pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:29.959355 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.959339 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/853903d4-a52c-4bc3-a1ae-20a096602158-cert\") pod \"ingress-canary-c64cd\" (UID: \"853903d4-a52c-4bc3-a1ae-20a096602158\") " pod="openshift-ingress-canary/ingress-canary-c64cd" Jul 10 16:55:29.965318 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.965294 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmpcz\" (UniqueName: \"kubernetes.io/projected/f1c389de-199e-468c-851d-1a5d8a9e51fe-kube-api-access-rmpcz\") pod \"dns-default-m5jwp\" (UID: \"f1c389de-199e-468c-851d-1a5d8a9e51fe\") " pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:29.965468 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:29.965452 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfpkd\" (UniqueName: \"kubernetes.io/projected/853903d4-a52c-4bc3-a1ae-20a096602158-kube-api-access-hfpkd\") pod \"ingress-canary-c64cd\" (UID: \"853903d4-a52c-4bc3-a1ae-20a096602158\") " pod="openshift-ingress-canary/ingress-canary-c64cd" Jul 10 16:55:30.022388 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.022360 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-j4xzk" Jul 10 16:55:30.045849 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.045814 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" Jul 10 16:55:30.052291 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.052242 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" Jul 10 16:55:30.057534 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.057509 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfr45\" (UniqueName: \"kubernetes.io/projected/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-kube-api-access-jfr45\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:30.057638 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.057543 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:30.057638 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.057563 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-crio-socket\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:30.057718 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.057645 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-crio-socket\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:30.057824 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.057798 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:30.057861 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.057848 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-data-volume\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:30.058082 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.058068 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:30.058238 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.058208 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-data-volume\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:30.060165 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.060148 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:30.071400 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.071369 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfr45\" (UniqueName: \"kubernetes.io/projected/793fd3dd-65e8-4ebf-87b1-8d5a9667b139-kube-api-access-jfr45\") pod \"insights-runtime-extractor-n4jsc\" (UID: \"793fd3dd-65e8-4ebf-87b1-8d5a9667b139\") " pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:30.096468 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.096430 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c64cd" Jul 10 16:55:30.109025 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.108997 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:30.203630 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.203539 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n4jsc" Jul 10 16:55:30.296171 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.296131 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5"] Jul 10 16:55:30.300454 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.300385 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b"] Jul 10 16:55:30.301078 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:55:30.301052 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf032d76f_2506_46c0_ab02_2bfa01906b9a.slice/crio-be8dc09830fe10fc138be9791265a6f0c5e4efc88b745d104fc70a3809979e59 WatchSource:0}: Error finding container be8dc09830fe10fc138be9791265a6f0c5e4efc88b745d104fc70a3809979e59: Status 404 returned error can't find the container with id be8dc09830fe10fc138be9791265a6f0c5e4efc88b745d104fc70a3809979e59 Jul 10 16:55:30.305230 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:55:30.305203 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b67970_933f_4673_b9f3_1dcdf445dcf1.slice/crio-95e7704b027c6a639d597e36c003fc0c839fa62c5fff7f29e0c782d6ed54f365 WatchSource:0}: Error finding container 95e7704b027c6a639d597e36c003fc0c839fa62c5fff7f29e0c782d6ed54f365: Status 404 returned error can't find the container with id 95e7704b027c6a639d597e36c003fc0c839fa62c5fff7f29e0c782d6ed54f365 Jul 10 16:55:30.311970 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.310694 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-j4xzk"] Jul 10 16:55:30.314053 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.313933 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c64cd"] Jul 10 16:55:30.323022 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:55:30.322985 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473995a2_4644_48b2_bb05_cc5aadfcc990.slice/crio-9e4a168b7ec22fee9daae0b6d91fc27aa56d3c9ac36c801e85b2c9387e661841 WatchSource:0}: Error finding container 9e4a168b7ec22fee9daae0b6d91fc27aa56d3c9ac36c801e85b2c9387e661841: Status 404 returned error can't find the container with id 9e4a168b7ec22fee9daae0b6d91fc27aa56d3c9ac36c801e85b2c9387e661841 Jul 10 16:55:30.324051 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:55:30.323901 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853903d4_a52c_4bc3_a1ae_20a096602158.slice/crio-49f557a8e50e9ce0fd96914a388a99921731dbd13542964fdbcec43f89257ba9 WatchSource:0}: Error finding container 49f557a8e50e9ce0fd96914a388a99921731dbd13542964fdbcec43f89257ba9: Status 404 returned error can't find the container with id 49f557a8e50e9ce0fd96914a388a99921731dbd13542964fdbcec43f89257ba9 Jul 10 16:55:30.334501 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.334460 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m5jwp"] Jul 10 16:55:30.338724 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:55:30.338699 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c389de_199e_468c_851d_1a5d8a9e51fe.slice/crio-1ad602dbb07a0c837e8ba7a2846b4c35ead749cefb633143fff9a2619ddc8795 WatchSource:0}: Error finding container 1ad602dbb07a0c837e8ba7a2846b4c35ead749cefb633143fff9a2619ddc8795: Status 404 returned error can't find the container with id 1ad602dbb07a0c837e8ba7a2846b4c35ead749cefb633143fff9a2619ddc8795 Jul 10 16:55:30.387838 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.387685 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n4jsc"] Jul 10 16:55:30.390191 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:55:30.390162 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793fd3dd_65e8_4ebf_87b1_8d5a9667b139.slice/crio-33d639d989c7c51b7dfd7af306c21e38f39ee44fb5b850118b976f625ab5d2bb WatchSource:0}: Error finding container 33d639d989c7c51b7dfd7af306c21e38f39ee44fb5b850118b976f625ab5d2bb: Status 404 returned error can't find the container with id 33d639d989c7c51b7dfd7af306c21e38f39ee44fb5b850118b976f625ab5d2bb Jul 10 16:55:30.947546 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:30.947510 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trj2g" Jul 10 16:55:31.011092 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:31.011010 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-j4xzk" event={"ID":"473995a2-4644-48b2-bb05-cc5aadfcc990","Type":"ContainerStarted","Data":"9e4a168b7ec22fee9daae0b6d91fc27aa56d3c9ac36c801e85b2c9387e661841"} Jul 10 16:55:31.012849 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:31.012768 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" event={"ID":"e0b67970-933f-4673-b9f3-1dcdf445dcf1","Type":"ContainerStarted","Data":"95e7704b027c6a639d597e36c003fc0c839fa62c5fff7f29e0c782d6ed54f365"} Jul 10 16:55:31.015971 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:31.015943 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m5jwp" event={"ID":"f1c389de-199e-468c-851d-1a5d8a9e51fe","Type":"ContainerStarted","Data":"1ad602dbb07a0c837e8ba7a2846b4c35ead749cefb633143fff9a2619ddc8795"} Jul 10 16:55:31.017682 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:31.017653 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c64cd" event={"ID":"853903d4-a52c-4bc3-a1ae-20a096602158","Type":"ContainerStarted","Data":"49f557a8e50e9ce0fd96914a388a99921731dbd13542964fdbcec43f89257ba9"} Jul 10 16:55:31.019765 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:31.019737 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n4jsc" event={"ID":"793fd3dd-65e8-4ebf-87b1-8d5a9667b139","Type":"ContainerStarted","Data":"33d639d989c7c51b7dfd7af306c21e38f39ee44fb5b850118b976f625ab5d2bb"} Jul 10 16:55:31.021144 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:31.021118 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" event={"ID":"f032d76f-2506-46c0-ab02-2bfa01906b9a","Type":"ContainerStarted","Data":"be8dc09830fe10fc138be9791265a6f0c5e4efc88b745d104fc70a3809979e59"} Jul 10 16:55:36.309683 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.309644 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs\") pod \"network-metrics-daemon-6cldf\" (UID: \"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c\") " pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:55:36.310252 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.309699 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5kn\" (UniqueName: \"kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn\") pod \"network-check-target-8gqbc\" (UID: \"832da8b4-c715-4416-9881-77c2baa30314\") " pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:55:36.311999 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.311975 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 16:55:36.312110 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.311975 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 16:55:36.323063 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.323031 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 16:55:36.323520 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.323493 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c-metrics-certs\") pod \"network-metrics-daemon-6cldf\" (UID: \"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c\") " pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:55:36.334629 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.334599 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l5kn\" (UniqueName: \"kubernetes.io/projected/832da8b4-c715-4416-9881-77c2baa30314-kube-api-access-9l5kn\") pod \"network-check-target-8gqbc\" (UID: \"832da8b4-c715-4416-9881-77c2baa30314\") " pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:55:36.389687 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.389652 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jn8d4\"" Jul 10 16:55:36.398468 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.398440 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:55:36.410296 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.410253 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:55:36.412917 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.412903 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 16:55:36.423466 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.423447 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3390d32-503c-4e8f-ab8d-c1b3881f2df9-original-pull-secret\") pod \"global-pull-secret-syncer-rf7b9\" (UID: \"e3390d32-503c-4e8f-ab8d-c1b3881f2df9\") " pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:55:36.492046 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.492014 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdmq6\"" Jul 10 16:55:36.493870 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.493848 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf7b9" Jul 10 16:55:36.500175 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:36.500155 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cldf" Jul 10 16:55:39.721403 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:39.721370 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rf7b9"] Jul 10 16:55:40.067484 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:40.067353 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rf7b9" event={"ID":"e3390d32-503c-4e8f-ab8d-c1b3881f2df9","Type":"ContainerStarted","Data":"0c6d97df78797e8ee52d61d1739dd1be5d869d5360304f52b333cfc995f749b8"} Jul 10 16:55:40.099213 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:40.098187 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-j4xzk" podStartSLOduration=2.3755643 podStartE2EDuration="11.098166221s" totalImagesPullingTime="8.722601921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:40.097535666 +0000 UTC m=+69.984827009" watchObservedRunningTime="2026-07-10 16:55:40.098166221 +0000 UTC m=+69.985457543" Jul 10 16:55:40.131262 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:40.131151 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6cldf"] Jul 10 16:55:40.134951 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:55:40.134916 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a7cd3c_d5fa_4909_9a2e_8f8eaaab4f6c.slice/crio-641e40afbd0ed614f22a4c689eceaba98edeea1f1af1862c59a34b9de0d1832e WatchSource:0}: Error finding container 641e40afbd0ed614f22a4c689eceaba98edeea1f1af1862c59a34b9de0d1832e: Status 404 returned error can't find the container with id 641e40afbd0ed614f22a4c689eceaba98edeea1f1af1862c59a34b9de0d1832e Jul 10 16:55:40.141146 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:40.138618 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8gqbc"] Jul 10 16:55:41.075668 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.075618 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m5jwp" event={"ID":"f1c389de-199e-468c-851d-1a5d8a9e51fe","Type":"ContainerStarted","Data":"46d97bece44fa9073ce7879f5e4a901d8eebb2692a86ec5ef94fed95cdb277a1"} Jul 10 16:55:41.076115 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.075684 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m5jwp" event={"ID":"f1c389de-199e-468c-851d-1a5d8a9e51fe","Type":"ContainerStarted","Data":"351ca5e085277f5ff848d379a544adf941d1d1c00711c6ddb31bcd17947c11ef"} Jul 10 16:55:41.076115 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.075800 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:41.078845 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.078489 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c64cd" event={"ID":"853903d4-a52c-4bc3-a1ae-20a096602158","Type":"ContainerStarted","Data":"b71a6d693b4459a1324b37464872d2375994a99733995e9a63b60c37b6cc3484"} Jul 10 16:55:41.080246 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.080220 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6cldf" event={"ID":"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c","Type":"ContainerStarted","Data":"641e40afbd0ed614f22a4c689eceaba98edeea1f1af1862c59a34b9de0d1832e"} Jul 10 16:55:41.082411 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.082329 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n4jsc" event={"ID":"793fd3dd-65e8-4ebf-87b1-8d5a9667b139","Type":"ContainerStarted","Data":"c64080413f210c5e163ba1968ccdcd71572600950fa9cb8b520836bd92eb281a"} Jul 10 16:55:41.082411 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.082372 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n4jsc" event={"ID":"793fd3dd-65e8-4ebf-87b1-8d5a9667b139","Type":"ContainerStarted","Data":"50d8373c297b2f8fa6eec3df2b6232e9567f3941850fb32b2d6d56252b9dc626"} Jul 10 16:55:41.085228 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.085207 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" event={"ID":"f032d76f-2506-46c0-ab02-2bfa01906b9a","Type":"ContainerStarted","Data":"02c8f9aa43dc0b14df90409bd6f8f9faf86f216f1471521b075219d91df28cd9"} Jul 10 16:55:41.085355 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.085240 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" event={"ID":"f032d76f-2506-46c0-ab02-2bfa01906b9a","Type":"ContainerStarted","Data":"cf271b54413757767adc7702807b0a00e04d0aab4da5621cb2ebea27f6173f00"} Jul 10 16:55:41.085355 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.085256 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" event={"ID":"f032d76f-2506-46c0-ab02-2bfa01906b9a","Type":"ContainerStarted","Data":"39a64d3fc77507af73ab02ea50483a99907868b3b0b2092be033d0dfa95553da"} Jul 10 16:55:41.087189 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.087136 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8gqbc" event={"ID":"832da8b4-c715-4416-9881-77c2baa30314","Type":"ContainerStarted","Data":"1d0476a8c36bd5cc7ea2a04a8e14bb4e642920eb93e80c60df8ed31acf72537b"} Jul 10 16:55:41.090053 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.089613 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-j4xzk" event={"ID":"473995a2-4644-48b2-bb05-cc5aadfcc990","Type":"ContainerStarted","Data":"0b82c950c21b3b06d4eae636956f605c590feefed8eef293a271641707a420e7"} Jul 10 16:55:41.092171 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.091693 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" event={"ID":"e0b67970-933f-4673-b9f3-1dcdf445dcf1","Type":"ContainerStarted","Data":"9622164d38f3d3f205415a2757cab981c8d55a5029295c8cc48a84326efa29c4"} Jul 10 16:55:41.094614 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.094594 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" Jul 10 16:55:41.099405 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.099317 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m5jwp" podStartSLOduration=2.851542812 podStartE2EDuration="12.099299778s" totalImagesPullingTime="9.247756966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:41.097834313 +0000 UTC m=+70.985125659" watchObservedRunningTime="2026-07-10 16:55:41.099299778 +0000 UTC m=+70.986591090" Jul 10 16:55:41.100156 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.100133 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" Jul 10 16:55:41.121813 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.121755 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c64cd" podStartSLOduration=2.859016448 podStartE2EDuration="12.121736513s" totalImagesPullingTime="9.262720065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:41.121234406 +0000 UTC m=+71.008525739" watchObservedRunningTime="2026-07-10 16:55:41.121736513 +0000 UTC m=+71.009027833" Jul 10 16:55:41.138266 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.138219 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cb47698f9-fd57b" podStartSLOduration=2.508772842 podStartE2EDuration="12.138206203s" totalImagesPullingTime="9.629433361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:41.137331488 +0000 UTC m=+71.024622808" watchObservedRunningTime="2026-07-10 16:55:41.138206203 +0000 UTC m=+71.025497522" Jul 10 16:55:41.160929 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.159750 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54db746db9-lwpz5" podStartSLOduration=3.415496648 podStartE2EDuration="12.159729909s" totalImagesPullingTime="8.744233261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:41.158005437 +0000 UTC m=+71.045296782" watchObservedRunningTime="2026-07-10 16:55:41.159729909 +0000 UTC m=+71.047021230" Jul 10 16:55:41.202184 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.202152 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z4mpd"] Jul 10 16:55:41.205416 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.205392 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.208074 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.208049 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 16:55:41.208213 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.208054 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 16:55:41.208444 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.208407 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 16:55:41.208713 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.208677 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 16:55:41.208713 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.208707 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 16:55:41.208865 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.208783 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kbv2t\"" Jul 10 16:55:41.208865 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.208829 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 16:55:41.250572 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.250539 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwwgw\" (UniqueName: \"kubernetes.io/projected/ba1d7c90-8728-431c-9e58-926cc6700a1a-kube-api-access-hwwgw\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.250743 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.250585 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba1d7c90-8728-431c-9e58-926cc6700a1a-root\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.250743 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.250610 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.250743 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.250640 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-tls\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.250743 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.250660 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.250743 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.250686 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-textfile\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.250743 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.250739 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba1d7c90-8728-431c-9e58-926cc6700a1a-sys\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.251046 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.250774 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-wtmp\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.251046 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.250794 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba1d7c90-8728-431c-9e58-926cc6700a1a-metrics-client-ca\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352072 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba1d7c90-8728-431c-9e58-926cc6700a1a-sys\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352132 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-wtmp\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352162 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba1d7c90-8728-431c-9e58-926cc6700a1a-metrics-client-ca\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352194 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwwgw\" (UniqueName: \"kubernetes.io/projected/ba1d7c90-8728-431c-9e58-926cc6700a1a-kube-api-access-hwwgw\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352219 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba1d7c90-8728-431c-9e58-926cc6700a1a-root\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352248 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba1d7c90-8728-431c-9e58-926cc6700a1a-sys\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352246 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352321 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-tls\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352343 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-wtmp\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352352 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352421 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-textfile\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:41.352436 2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:55:41.352508 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-tls podName:ba1d7c90-8728-431c-9e58-926cc6700a1a nodeName:}" failed. No retries permitted until 2026-07-10 16:55:41.852486902 +0000 UTC m=+71.739778201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-tls") pod "node-exporter-z4mpd" (UID: "ba1d7c90-8728-431c-9e58-926cc6700a1a") : secret "node-exporter-tls" not found Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352814 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-textfile\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352864 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.352929 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba1d7c90-8728-431c-9e58-926cc6700a1a-root\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.353500 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.353419 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba1d7c90-8728-431c-9e58-926cc6700a1a-metrics-client-ca\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.359329 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.359262 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.363221 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.363179 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwwgw\" (UniqueName: \"kubernetes.io/projected/ba1d7c90-8728-431c-9e58-926cc6700a1a-kube-api-access-hwwgw\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.856593 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.856528 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-tls\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:41.860468 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:41.860435 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba1d7c90-8728-431c-9e58-926cc6700a1a-node-exporter-tls\") pod \"node-exporter-z4mpd\" (UID: \"ba1d7c90-8728-431c-9e58-926cc6700a1a\") " pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:42.100012 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:42.099976 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6cldf" event={"ID":"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c","Type":"ContainerStarted","Data":"88cc56745bd1a761fbc58260d7218578cbccdfacbe7648bc8da619a2ae00078b"} Jul 10 16:55:42.100491 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:42.100033 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6cldf" event={"ID":"f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c","Type":"ContainerStarted","Data":"c7b4d8d0866990a317db3f742d45d237d902df50dcb34319a781b531fb31e4a0"} Jul 10 16:55:42.118337 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:42.117939 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z4mpd" Jul 10 16:55:42.120790 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:42.120217 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6cldf" podStartSLOduration=69.945484214 podStartE2EDuration="1m11.120195771s" totalImagesPullingTime="1.174711557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:42.11912334 +0000 UTC m=+72.006414664" watchObservedRunningTime="2026-07-10 16:55:42.120195771 +0000 UTC m=+72.007487272" Jul 10 16:55:42.131783 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:55:42.131749 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1d7c90_8728_431c_9e58_926cc6700a1a.slice/crio-73094fcd3151dad1d96a9717196063acdd92d10d0f2729e2abb3a94c04914cbe WatchSource:0}: Error finding container 73094fcd3151dad1d96a9717196063acdd92d10d0f2729e2abb3a94c04914cbe: Status 404 returned error can't find the container with id 73094fcd3151dad1d96a9717196063acdd92d10d0f2729e2abb3a94c04914cbe Jul 10 16:55:43.104375 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:43.104332 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4mpd" event={"ID":"ba1d7c90-8728-431c-9e58-926cc6700a1a","Type":"ContainerStarted","Data":"73094fcd3151dad1d96a9717196063acdd92d10d0f2729e2abb3a94c04914cbe"} Jul 10 16:55:45.576745 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.576709 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b4d5c6978-hr8v5"] Jul 10 16:55:45.579775 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.579754 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.583448 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.583419 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 16:55:45.583577 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.583477 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-x6jcc\"" Jul 10 16:55:45.583577 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.583510 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 16:55:45.583692 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.583420 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d0qfps5pg4oe1\"" Jul 10 16:55:45.583692 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.583424 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 16:55:45.583692 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.583671 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 16:55:45.591543 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.591494 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b4d5c6978-hr8v5"] Jul 10 16:55:45.691485 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.691448 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/06a4fc65-bef5-44e1-b862-48d154fb2fc4-secret-metrics-server-client-certs\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.691485 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.691491 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xktqj\" (UniqueName: \"kubernetes.io/projected/06a4fc65-bef5-44e1-b862-48d154fb2fc4-kube-api-access-xktqj\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.691746 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.691518 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/06a4fc65-bef5-44e1-b862-48d154fb2fc4-client-ca-bundle\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.691746 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.691620 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/06a4fc65-bef5-44e1-b862-48d154fb2fc4-audit-log\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.691746 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.691696 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/06a4fc65-bef5-44e1-b862-48d154fb2fc4-metrics-server-audit-profiles\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.691897 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.691761 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/06a4fc65-bef5-44e1-b862-48d154fb2fc4-secret-metrics-server-tls\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.691897 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.691827 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a4fc65-bef5-44e1-b862-48d154fb2fc4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.792718 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.792667 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/06a4fc65-bef5-44e1-b862-48d154fb2fc4-metrics-server-audit-profiles\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.792927 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.792729 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/06a4fc65-bef5-44e1-b862-48d154fb2fc4-secret-metrics-server-tls\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.792927 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.792768 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a4fc65-bef5-44e1-b862-48d154fb2fc4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.792927 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.792800 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/06a4fc65-bef5-44e1-b862-48d154fb2fc4-secret-metrics-server-client-certs\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.792927 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.792826 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xktqj\" (UniqueName: \"kubernetes.io/projected/06a4fc65-bef5-44e1-b862-48d154fb2fc4-kube-api-access-xktqj\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.792927 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.792853 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/06a4fc65-bef5-44e1-b862-48d154fb2fc4-client-ca-bundle\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.792927 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.792885 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/06a4fc65-bef5-44e1-b862-48d154fb2fc4-audit-log\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.793711 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.793684 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a4fc65-bef5-44e1-b862-48d154fb2fc4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.794255 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.794232 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/06a4fc65-bef5-44e1-b862-48d154fb2fc4-metrics-server-audit-profiles\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.794403 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.794358 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/06a4fc65-bef5-44e1-b862-48d154fb2fc4-audit-log\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.795794 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.795769 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/06a4fc65-bef5-44e1-b862-48d154fb2fc4-secret-metrics-server-tls\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.795918 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.795907 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/06a4fc65-bef5-44e1-b862-48d154fb2fc4-secret-metrics-server-client-certs\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.796716 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.796689 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/06a4fc65-bef5-44e1-b862-48d154fb2fc4-client-ca-bundle\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.802034 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.802007 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xktqj\" (UniqueName: \"kubernetes.io/projected/06a4fc65-bef5-44e1-b862-48d154fb2fc4-kube-api-access-xktqj\") pod \"metrics-server-7b4d5c6978-hr8v5\" (UID: \"06a4fc65-bef5-44e1-b862-48d154fb2fc4\") " pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:45.891607 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:45.891514 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:55:46.499168 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:46.499041 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b4d5c6978-hr8v5"] Jul 10 16:55:46.501728 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:55:46.501700 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a4fc65_bef5_44e1_b862_48d154fb2fc4.slice/crio-f90801f445c7cc9ba555102f433aa1f632aa352362f8c51fc3cc760f69b8aaad WatchSource:0}: Error finding container f90801f445c7cc9ba555102f433aa1f632aa352362f8c51fc3cc760f69b8aaad: Status 404 returned error can't find the container with id f90801f445c7cc9ba555102f433aa1f632aa352362f8c51fc3cc760f69b8aaad Jul 10 16:55:47.122972 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:47.122930 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n4jsc" event={"ID":"793fd3dd-65e8-4ebf-87b1-8d5a9667b139","Type":"ContainerStarted","Data":"34cb1b9bc8fd13ddfb5d48bf6e04d7bdf52c768c1ce71b467bb56a0d249beac2"} Jul 10 16:55:47.124590 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:47.124563 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rf7b9" event={"ID":"e3390d32-503c-4e8f-ab8d-c1b3881f2df9","Type":"ContainerStarted","Data":"6b86601287718dace2b2d46f9dc156c0df45447af60bc7e97d34f6721c664f11"} Jul 10 16:55:47.126341 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:47.126313 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8gqbc" event={"ID":"832da8b4-c715-4416-9881-77c2baa30314","Type":"ContainerStarted","Data":"963abfb9a553e254c90bcd5dbaaabe5293d36853e79b4dafa2cafc2dde9a77e1"} Jul 10 16:55:47.126475 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:47.126459 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:55:47.127624 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:47.127601 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" event={"ID":"06a4fc65-bef5-44e1-b862-48d154fb2fc4","Type":"ContainerStarted","Data":"f90801f445c7cc9ba555102f433aa1f632aa352362f8c51fc3cc760f69b8aaad"} Jul 10 16:55:47.129183 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:47.129154 2567 generic.go:356] "Generic (PLEG): container finished" podID="ba1d7c90-8728-431c-9e58-926cc6700a1a" containerID="3cf86b4e5a021ece0900c5645c25f844ae6aa23447147a3187399b8cc0125824" exitCode=0 Jul 10 16:55:47.129309 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:47.129196 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4mpd" event={"ID":"ba1d7c90-8728-431c-9e58-926cc6700a1a","Type":"ContainerDied","Data":"3cf86b4e5a021ece0900c5645c25f844ae6aa23447147a3187399b8cc0125824"} Jul 10 16:55:47.155356 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:47.155115 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n4jsc" podStartSLOduration=2.4307262 podStartE2EDuration="18.155096856s" totalImagesPullingTime="15.724370656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:55:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:47.153516409 +0000 UTC m=+77.040807731" watchObservedRunningTime="2026-07-10 16:55:47.155096856 +0000 UTC m=+77.042388177" Jul 10 16:55:47.181318 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:47.181245 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rf7b9" podStartSLOduration=69.760268853 podStartE2EDuration="1m16.181227383s" totalImagesPullingTime="6.42095853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:47.178838213 +0000 UTC m=+77.066129558" watchObservedRunningTime="2026-07-10 16:55:47.181227383 +0000 UTC m=+77.068518699" Jul 10 16:55:47.200807 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:47.200720 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8gqbc" podStartSLOduration=69.990454813 podStartE2EDuration="1m16.20070042s" totalImagesPullingTime="6.210245607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:47.200688957 +0000 UTC m=+77.087980283" watchObservedRunningTime="2026-07-10 16:55:47.20070042 +0000 UTC m=+77.087991741" Jul 10 16:55:48.134018 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:48.133977 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" event={"ID":"06a4fc65-bef5-44e1-b862-48d154fb2fc4","Type":"ContainerStarted","Data":"6a219b289ebacb18d74f237a2c3d7c981f629816171d520bc7f6e617ff9d3347"} Jul 10 16:55:48.136223 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:48.136191 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4mpd" event={"ID":"ba1d7c90-8728-431c-9e58-926cc6700a1a","Type":"ContainerStarted","Data":"8b91c55b0ca2efbc3f1c9e6ec80b2b68bb58da95377ca604db730c2e43eac152"} Jul 10 16:55:48.136352 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:48.136241 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4mpd" event={"ID":"ba1d7c90-8728-431c-9e58-926cc6700a1a","Type":"ContainerStarted","Data":"345cbc5f40b6b61db39146df6f08723853093d37d6f18e4b8a6aaa0dad20c6c4"} Jul 10 16:55:48.155683 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:48.155634 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" podStartSLOduration=1.706804477 podStartE2EDuration="3.155615692s" totalImagesPullingTime="1.448811215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:48.154484118 +0000 UTC m=+78.041775438" watchObservedRunningTime="2026-07-10 16:55:48.155615692 +0000 UTC m=+78.042907013" Jul 10 16:55:48.178795 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:48.178751 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z4mpd" podStartSLOduration=2.8832548989999998 podStartE2EDuration="7.178738697s" totalImagesPullingTime="4.218278516s" totalInitContainerRuntime="77.205282ms" isStatefulPod=true podCreationTimestamp="2026-07-10 16:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:48.177514242 +0000 UTC m=+78.064805562" watchObservedRunningTime="2026-07-10 16:55:48.178738697 +0000 UTC m=+78.066030016" Jul 10 16:55:51.103339 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:51.103184 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m5jwp" Jul 10 16:55:52.163750 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.163719 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67c66cc96d-njsfr"] Jul 10 16:55:52.166176 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.166159 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.169788 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.169764 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 16:55:52.169932 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.169790 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 16:55:52.169932 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.169868 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 16:55:52.169932 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.169880 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 16:55:52.169932 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.169882 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6jlpk\"" Jul 10 16:55:52.169932 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.169913 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 16:55:52.169932 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.169933 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 16:55:52.171764 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.171744 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 16:55:52.174546 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.174529 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 16:55:52.176754 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.176735 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67c66cc96d-njsfr"] Jul 10 16:55:52.239520 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.239493 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-trusted-ca-bundle\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.239637 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.239524 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d9th\" (UniqueName: \"kubernetes.io/projected/a0095721-add4-4426-aa77-d00ecf8539bf-kube-api-access-4d9th\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.239637 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.239549 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-console-config\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.239637 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.239619 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0095721-add4-4426-aa77-d00ecf8539bf-console-serving-cert\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.239757 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.239644 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-oauth-serving-cert\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.239757 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.239671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0095721-add4-4426-aa77-d00ecf8539bf-console-oauth-config\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.239757 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.239688 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-service-ca\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.340782 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.340748 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-trusted-ca-bundle\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.340904 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.340786 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d9th\" (UniqueName: \"kubernetes.io/projected/a0095721-add4-4426-aa77-d00ecf8539bf-kube-api-access-4d9th\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.340904 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.340809 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-console-config\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.340904 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.340833 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0095721-add4-4426-aa77-d00ecf8539bf-console-serving-cert\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.341082 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.340953 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-oauth-serving-cert\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.341082 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.341007 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0095721-add4-4426-aa77-d00ecf8539bf-console-oauth-config\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.341082 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.341033 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-service-ca\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.342534 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.342504 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-oauth-serving-cert\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.342534 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.342513 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-console-config\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.343650 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.343579 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-service-ca\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.344510 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.343812 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-trusted-ca-bundle\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.345015 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.344992 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0095721-add4-4426-aa77-d00ecf8539bf-console-serving-cert\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.345101 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.345018 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0095721-add4-4426-aa77-d00ecf8539bf-console-oauth-config\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.361612 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.361582 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d9th\" (UniqueName: \"kubernetes.io/projected/a0095721-add4-4426-aa77-d00ecf8539bf-kube-api-access-4d9th\") pod \"console-67c66cc96d-njsfr\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.475863 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.475826 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:55:52.599881 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:52.599852 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67c66cc96d-njsfr"] Jul 10 16:55:52.602296 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:55:52.602252 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0095721_add4_4426_aa77_d00ecf8539bf.slice/crio-c96e6cd7aa81d2e0c28b364bdcdd347a00fd93981adc8cadc9977bbe5d643770 WatchSource:0}: Error finding container c96e6cd7aa81d2e0c28b364bdcdd347a00fd93981adc8cadc9977bbe5d643770: Status 404 returned error can't find the container with id c96e6cd7aa81d2e0c28b364bdcdd347a00fd93981adc8cadc9977bbe5d643770 Jul 10 16:55:53.156426 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:53.156385 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c66cc96d-njsfr" event={"ID":"a0095721-add4-4426-aa77-d00ecf8539bf","Type":"ContainerStarted","Data":"c96e6cd7aa81d2e0c28b364bdcdd347a00fd93981adc8cadc9977bbe5d643770"} Jul 10 16:55:56.170363 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:56.170326 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c66cc96d-njsfr" event={"ID":"a0095721-add4-4426-aa77-d00ecf8539bf","Type":"ContainerStarted","Data":"154eb3aba601cb7409543d3cbe1678b32d698d5b469942dcf5dbee519ff52375"} Jul 10 16:55:56.190217 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:55:56.190171 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67c66cc96d-njsfr" podStartSLOduration=1.219543741 podStartE2EDuration="4.190156028s" totalImagesPullingTime="2.970612287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:56.189071331 +0000 UTC m=+86.076362650" watchObservedRunningTime="2026-07-10 16:55:56.190156028 +0000 UTC m=+86.077447347" Jul 10 16:56:02.476296 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:56:02.476230 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:56:02.476709 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:56:02.476337 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:56:02.481135 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:56:02.481110 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:56:03.198339 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:56:03.198243 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:56:05.892326 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:56:05.892290 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:56:05.892785 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:56:05.892363 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:56:18.139121 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:56:18.139087 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8gqbc" Jul 10 16:56:25.897426 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:56:25.897389 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:56:25.902495 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:56:25.901148 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b4d5c6978-hr8v5" Jul 10 16:57:10.305988 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.305908 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-794b7dd9d7-9bkbj"] Jul 10 16:57:10.308118 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.308099 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.318420 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.318395 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794b7dd9d7-9bkbj"] Jul 10 16:57:10.418599 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.418565 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2f29d32-c014-427e-9275-788e2eb3d379-console-serving-cert\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.418757 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.418610 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-trusted-ca-bundle\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.418757 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.418640 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxk8x\" (UniqueName: \"kubernetes.io/projected/b2f29d32-c014-427e-9275-788e2eb3d379-kube-api-access-rxk8x\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.418757 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.418696 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2f29d32-c014-427e-9275-788e2eb3d379-console-oauth-config\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.418757 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.418745 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-service-ca\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.418890 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.418762 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-oauth-serving-cert\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.418890 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.418788 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-console-config\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.519835 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.519800 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2f29d32-c014-427e-9275-788e2eb3d379-console-oauth-config\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.520015 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.519845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-service-ca\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.520015 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.519863 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-oauth-serving-cert\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.520015 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.519891 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-console-config\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.520015 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.519930 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2f29d32-c014-427e-9275-788e2eb3d379-console-serving-cert\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.520015 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.519970 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-trusted-ca-bundle\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.520015 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.519994 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxk8x\" (UniqueName: \"kubernetes.io/projected/b2f29d32-c014-427e-9275-788e2eb3d379-kube-api-access-rxk8x\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.520754 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.520721 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-console-config\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.520891 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.520862 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-service-ca\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.521000 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.520975 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-oauth-serving-cert\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.521117 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.521096 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-trusted-ca-bundle\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.522490 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.522470 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2f29d32-c014-427e-9275-788e2eb3d379-console-oauth-config\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.522588 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.522540 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2f29d32-c014-427e-9275-788e2eb3d379-console-serving-cert\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.528318 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.528298 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxk8x\" (UniqueName: \"kubernetes.io/projected/b2f29d32-c014-427e-9275-788e2eb3d379-kube-api-access-rxk8x\") pod \"console-794b7dd9d7-9bkbj\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.616660 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.616584 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:10.742772 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:10.742741 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794b7dd9d7-9bkbj"] Jul 10 16:57:10.745613 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:57:10.745575 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f29d32_c014_427e_9275_788e2eb3d379.slice/crio-4c0738d90a861464445ed2bc1f7d655549b4097963e18704db0b10d37f286edc WatchSource:0}: Error finding container 4c0738d90a861464445ed2bc1f7d655549b4097963e18704db0b10d37f286edc: Status 404 returned error can't find the container with id 4c0738d90a861464445ed2bc1f7d655549b4097963e18704db0b10d37f286edc Jul 10 16:57:11.425562 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:11.425527 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794b7dd9d7-9bkbj" event={"ID":"b2f29d32-c014-427e-9275-788e2eb3d379","Type":"ContainerStarted","Data":"d1403391a4a5b1070c917358924723dc200b96f4093ac450eb117bcec5a8b0cb"} Jul 10 16:57:11.425932 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:11.425599 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794b7dd9d7-9bkbj" event={"ID":"b2f29d32-c014-427e-9275-788e2eb3d379","Type":"ContainerStarted","Data":"4c0738d90a861464445ed2bc1f7d655549b4097963e18704db0b10d37f286edc"} Jul 10 16:57:11.442489 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:11.442444 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-794b7dd9d7-9bkbj" podStartSLOduration=1.442431072 podStartE2EDuration="1.442431072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:57:11.441915157 +0000 UTC m=+161.329206478" watchObservedRunningTime="2026-07-10 16:57:11.442431072 +0000 UTC m=+161.329722391" Jul 10 16:57:20.617656 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:20.617568 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:20.618111 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:20.617892 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:20.622494 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:20.622473 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:21.467454 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:21.467417 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:57:21.512633 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:21.512600 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67c66cc96d-njsfr"] Jul 10 16:57:46.533007 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.532959 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67c66cc96d-njsfr" podUID="a0095721-add4-4426-aa77-d00ecf8539bf" containerName="console" containerID="cri-o://154eb3aba601cb7409543d3cbe1678b32d698d5b469942dcf5dbee519ff52375" gracePeriod=15 Jul 10 16:57:46.780154 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.780130 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:57:46.882467 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.882374 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0095721-add4-4426-aa77-d00ecf8539bf-console-oauth-config\") pod \"a0095721-add4-4426-aa77-d00ecf8539bf\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " Jul 10 16:57:46.882467 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.882425 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-trusted-ca-bundle\") pod \"a0095721-add4-4426-aa77-d00ecf8539bf\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " Jul 10 16:57:46.882467 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.882463 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-service-ca\") pod \"a0095721-add4-4426-aa77-d00ecf8539bf\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " Jul 10 16:57:46.882725 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.882506 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-console-config\") pod \"a0095721-add4-4426-aa77-d00ecf8539bf\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " Jul 10 16:57:46.882725 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.882536 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d9th\" (UniqueName: \"kubernetes.io/projected/a0095721-add4-4426-aa77-d00ecf8539bf-kube-api-access-4d9th\") pod \"a0095721-add4-4426-aa77-d00ecf8539bf\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " Jul 10 16:57:46.882725 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.882566 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0095721-add4-4426-aa77-d00ecf8539bf-console-serving-cert\") pod \"a0095721-add4-4426-aa77-d00ecf8539bf\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " Jul 10 16:57:46.882725 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.882592 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-oauth-serving-cert\") pod \"a0095721-add4-4426-aa77-d00ecf8539bf\" (UID: \"a0095721-add4-4426-aa77-d00ecf8539bf\") " Jul 10 16:57:46.883039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.882913 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-console-config" (OuterVolumeSpecName: "console-config") pod "a0095721-add4-4426-aa77-d00ecf8539bf" (UID: "a0095721-add4-4426-aa77-d00ecf8539bf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:57:46.883039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.882931 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a0095721-add4-4426-aa77-d00ecf8539bf" (UID: "a0095721-add4-4426-aa77-d00ecf8539bf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:57:46.883039 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.882988 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-service-ca" (OuterVolumeSpecName: "service-ca") pod "a0095721-add4-4426-aa77-d00ecf8539bf" (UID: "a0095721-add4-4426-aa77-d00ecf8539bf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:57:46.883220 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.883090 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a0095721-add4-4426-aa77-d00ecf8539bf" (UID: "a0095721-add4-4426-aa77-d00ecf8539bf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:57:46.884834 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.884808 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0095721-add4-4426-aa77-d00ecf8539bf-kube-api-access-4d9th" (OuterVolumeSpecName: "kube-api-access-4d9th") pod "a0095721-add4-4426-aa77-d00ecf8539bf" (UID: "a0095721-add4-4426-aa77-d00ecf8539bf"). InnerVolumeSpecName "kube-api-access-4d9th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:57:46.884924 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.884839 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0095721-add4-4426-aa77-d00ecf8539bf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a0095721-add4-4426-aa77-d00ecf8539bf" (UID: "a0095721-add4-4426-aa77-d00ecf8539bf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:57:46.884924 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.884854 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0095721-add4-4426-aa77-d00ecf8539bf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a0095721-add4-4426-aa77-d00ecf8539bf" (UID: "a0095721-add4-4426-aa77-d00ecf8539bf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:57:46.983724 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.983698 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0095721-add4-4426-aa77-d00ecf8539bf-console-serving-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:57:46.983724 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.983720 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-oauth-serving-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:57:46.983865 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.983731 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0095721-add4-4426-aa77-d00ecf8539bf-console-oauth-config\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:57:46.983865 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.983741 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-trusted-ca-bundle\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:57:46.983865 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.983749 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:57:46.983865 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.983757 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0095721-add4-4426-aa77-d00ecf8539bf-console-config\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:57:46.983865 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:46.983765 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d9th\" (UniqueName: \"kubernetes.io/projected/a0095721-add4-4426-aa77-d00ecf8539bf-kube-api-access-4d9th\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:57:47.545153 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:47.545120 2567 generic.go:356] "Generic (PLEG): container finished" podID="a0095721-add4-4426-aa77-d00ecf8539bf" containerID="154eb3aba601cb7409543d3cbe1678b32d698d5b469942dcf5dbee519ff52375" exitCode=0 Jul 10 16:57:47.545565 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:47.545169 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c66cc96d-njsfr" Jul 10 16:57:47.545565 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:47.545193 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c66cc96d-njsfr" event={"ID":"a0095721-add4-4426-aa77-d00ecf8539bf","Type":"ContainerDied","Data":"154eb3aba601cb7409543d3cbe1678b32d698d5b469942dcf5dbee519ff52375"} Jul 10 16:57:47.545565 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:47.545232 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c66cc96d-njsfr" event={"ID":"a0095721-add4-4426-aa77-d00ecf8539bf","Type":"ContainerDied","Data":"c96e6cd7aa81d2e0c28b364bdcdd347a00fd93981adc8cadc9977bbe5d643770"} Jul 10 16:57:47.545565 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:47.545251 2567 scope.go:117] "RemoveContainer" containerID="154eb3aba601cb7409543d3cbe1678b32d698d5b469942dcf5dbee519ff52375" Jul 10 16:57:47.553676 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:47.553659 2567 scope.go:117] "RemoveContainer" containerID="154eb3aba601cb7409543d3cbe1678b32d698d5b469942dcf5dbee519ff52375" Jul 10 16:57:47.553952 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:57:47.553935 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154eb3aba601cb7409543d3cbe1678b32d698d5b469942dcf5dbee519ff52375\": container with ID starting with 154eb3aba601cb7409543d3cbe1678b32d698d5b469942dcf5dbee519ff52375 not found: ID does not exist" containerID="154eb3aba601cb7409543d3cbe1678b32d698d5b469942dcf5dbee519ff52375" Jul 10 16:57:47.553996 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:47.553961 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154eb3aba601cb7409543d3cbe1678b32d698d5b469942dcf5dbee519ff52375"} err="failed to get container status \"154eb3aba601cb7409543d3cbe1678b32d698d5b469942dcf5dbee519ff52375\": rpc error: code = NotFound desc = could not find container \"154eb3aba601cb7409543d3cbe1678b32d698d5b469942dcf5dbee519ff52375\": container with ID starting with 154eb3aba601cb7409543d3cbe1678b32d698d5b469942dcf5dbee519ff52375 not found: ID does not exist" Jul 10 16:57:47.564723 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:47.564698 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67c66cc96d-njsfr"] Jul 10 16:57:47.569043 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:47.569015 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67c66cc96d-njsfr"] Jul 10 16:57:48.680475 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:57:48.680437 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0095721-add4-4426-aa77-d00ecf8539bf" path="/var/lib/kubelet/pods/a0095721-add4-4426-aa77-d00ecf8539bf/volumes" Jul 10 16:58:16.168876 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.168841 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66c876d8c5-xcqlw"] Jul 10 16:58:16.170157 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.170132 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0095721-add4-4426-aa77-d00ecf8539bf" containerName="console" Jul 10 16:58:16.170330 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.170318 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0095721-add4-4426-aa77-d00ecf8539bf" containerName="console" Jul 10 16:58:16.170594 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.170581 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0095721-add4-4426-aa77-d00ecf8539bf" containerName="console" Jul 10 16:58:16.173768 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.173740 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.182424 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.182399 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c876d8c5-xcqlw"] Jul 10 16:58:16.286546 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.286506 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-serving-cert\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.286546 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.286543 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-trusted-ca-bundle\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.286757 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.286568 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgwrl\" (UniqueName: \"kubernetes.io/projected/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-kube-api-access-hgwrl\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.286757 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.286586 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-config\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.286757 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.286642 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-oauth-config\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.286757 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.286681 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-service-ca\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.286757 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.286707 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-oauth-serving-cert\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.387546 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.387500 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-service-ca\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.387705 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.387561 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-oauth-serving-cert\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.387705 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.387598 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-serving-cert\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.387705 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.387624 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-trusted-ca-bundle\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.387705 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.387659 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgwrl\" (UniqueName: \"kubernetes.io/projected/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-kube-api-access-hgwrl\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.387705 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.387682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-config\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.387991 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.387717 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-oauth-config\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.388499 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.388409 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-service-ca\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.388499 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.388409 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-oauth-serving-cert\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.388499 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.388542 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-trusted-ca-bundle\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.388499 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.388542 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-config\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.390147 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.390115 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-serving-cert\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.390300 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.390261 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-oauth-config\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.395852 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.395831 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgwrl\" (UniqueName: \"kubernetes.io/projected/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-kube-api-access-hgwrl\") pod \"console-66c876d8c5-xcqlw\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.482801 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.482770 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:16.607739 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.607715 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c876d8c5-xcqlw"] Jul 10 16:58:16.610239 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:58:16.610209 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59a60cb_69e9_43d3_a4b8_1981412f2ee3.slice/crio-a9a12ddd6efea544933eb8b1c2f724d35de96d03deed622213e016716d1a19b3 WatchSource:0}: Error finding container a9a12ddd6efea544933eb8b1c2f724d35de96d03deed622213e016716d1a19b3: Status 404 returned error can't find the container with id a9a12ddd6efea544933eb8b1c2f724d35de96d03deed622213e016716d1a19b3 Jul 10 16:58:16.639443 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:16.639418 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c876d8c5-xcqlw" event={"ID":"c59a60cb-69e9-43d3-a4b8-1981412f2ee3","Type":"ContainerStarted","Data":"a9a12ddd6efea544933eb8b1c2f724d35de96d03deed622213e016716d1a19b3"} Jul 10 16:58:17.644730 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:17.644684 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c876d8c5-xcqlw" event={"ID":"c59a60cb-69e9-43d3-a4b8-1981412f2ee3","Type":"ContainerStarted","Data":"f6f2907695df39bff0f9b649c9ca6b84cf2fd466a6751925475ce238ee9f9873"} Jul 10 16:58:17.663330 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:17.663262 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66c876d8c5-xcqlw" podStartSLOduration=1.663247892 podStartE2EDuration="1.663247892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:58:17.661693057 +0000 UTC m=+227.548984400" watchObservedRunningTime="2026-07-10 16:58:17.663247892 +0000 UTC m=+227.550539212" Jul 10 16:58:26.483105 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:26.483068 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:26.483606 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:26.483134 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:26.487611 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:26.487589 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:26.680776 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:26.680745 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 16:58:26.727869 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:26.727841 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-794b7dd9d7-9bkbj"] Jul 10 16:58:51.750392 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:51.750249 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-794b7dd9d7-9bkbj" podUID="b2f29d32-c014-427e-9275-788e2eb3d379" containerName="console" containerID="cri-o://d1403391a4a5b1070c917358924723dc200b96f4093ac450eb117bcec5a8b0cb" gracePeriod=15 Jul 10 16:58:51.982786 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:51.982753 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:58:52.053443 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.053355 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-oauth-serving-cert\") pod \"b2f29d32-c014-427e-9275-788e2eb3d379\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " Jul 10 16:58:52.053443 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.053406 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-console-config\") pod \"b2f29d32-c014-427e-9275-788e2eb3d379\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " Jul 10 16:58:52.053678 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.053453 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2f29d32-c014-427e-9275-788e2eb3d379-console-oauth-config\") pod \"b2f29d32-c014-427e-9275-788e2eb3d379\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " Jul 10 16:58:52.053678 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.053625 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-trusted-ca-bundle\") pod \"b2f29d32-c014-427e-9275-788e2eb3d379\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " Jul 10 16:58:52.053678 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.053665 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxk8x\" (UniqueName: \"kubernetes.io/projected/b2f29d32-c014-427e-9275-788e2eb3d379-kube-api-access-rxk8x\") pod \"b2f29d32-c014-427e-9275-788e2eb3d379\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " Jul 10 16:58:52.053843 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.053689 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2f29d32-c014-427e-9275-788e2eb3d379-console-serving-cert\") pod \"b2f29d32-c014-427e-9275-788e2eb3d379\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " Jul 10 16:58:52.053843 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.053725 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-service-ca\") pod \"b2f29d32-c014-427e-9275-788e2eb3d379\" (UID: \"b2f29d32-c014-427e-9275-788e2eb3d379\") " Jul 10 16:58:52.053843 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.053783 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b2f29d32-c014-427e-9275-788e2eb3d379" (UID: "b2f29d32-c014-427e-9275-788e2eb3d379"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:58:52.053985 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.053930 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-console-config" (OuterVolumeSpecName: "console-config") pod "b2f29d32-c014-427e-9275-788e2eb3d379" (UID: "b2f29d32-c014-427e-9275-788e2eb3d379"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:58:52.053985 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.053965 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-oauth-serving-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:58:52.054066 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.054039 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b2f29d32-c014-427e-9275-788e2eb3d379" (UID: "b2f29d32-c014-427e-9275-788e2eb3d379"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:58:52.054214 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.054196 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-service-ca" (OuterVolumeSpecName: "service-ca") pod "b2f29d32-c014-427e-9275-788e2eb3d379" (UID: "b2f29d32-c014-427e-9275-788e2eb3d379"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:58:52.055769 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.055739 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f29d32-c014-427e-9275-788e2eb3d379-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b2f29d32-c014-427e-9275-788e2eb3d379" (UID: "b2f29d32-c014-427e-9275-788e2eb3d379"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:58:52.055877 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.055783 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f29d32-c014-427e-9275-788e2eb3d379-kube-api-access-rxk8x" (OuterVolumeSpecName: "kube-api-access-rxk8x") pod "b2f29d32-c014-427e-9275-788e2eb3d379" (UID: "b2f29d32-c014-427e-9275-788e2eb3d379"). InnerVolumeSpecName "kube-api-access-rxk8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:58:52.055877 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.055810 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f29d32-c014-427e-9275-788e2eb3d379-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b2f29d32-c014-427e-9275-788e2eb3d379" (UID: "b2f29d32-c014-427e-9275-788e2eb3d379"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:58:52.155349 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.155302 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-console-config\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:58:52.155349 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.155340 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2f29d32-c014-427e-9275-788e2eb3d379-console-oauth-config\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:58:52.155349 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.155353 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-trusted-ca-bundle\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:58:52.155578 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.155366 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxk8x\" (UniqueName: \"kubernetes.io/projected/b2f29d32-c014-427e-9275-788e2eb3d379-kube-api-access-rxk8x\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:58:52.155578 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.155378 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2f29d32-c014-427e-9275-788e2eb3d379-console-serving-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:58:52.155578 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.155389 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f29d32-c014-427e-9275-788e2eb3d379-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 16:58:52.761842 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.761809 2567 generic.go:356] "Generic (PLEG): container finished" podID="b2f29d32-c014-427e-9275-788e2eb3d379" containerID="d1403391a4a5b1070c917358924723dc200b96f4093ac450eb117bcec5a8b0cb" exitCode=0 Jul 10 16:58:52.762297 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.761853 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794b7dd9d7-9bkbj" Jul 10 16:58:52.762297 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.761867 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794b7dd9d7-9bkbj" event={"ID":"b2f29d32-c014-427e-9275-788e2eb3d379","Type":"ContainerDied","Data":"d1403391a4a5b1070c917358924723dc200b96f4093ac450eb117bcec5a8b0cb"} Jul 10 16:58:52.762297 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.761908 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794b7dd9d7-9bkbj" event={"ID":"b2f29d32-c014-427e-9275-788e2eb3d379","Type":"ContainerDied","Data":"4c0738d90a861464445ed2bc1f7d655549b4097963e18704db0b10d37f286edc"} Jul 10 16:58:52.762297 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.761927 2567 scope.go:117] "RemoveContainer" containerID="d1403391a4a5b1070c917358924723dc200b96f4093ac450eb117bcec5a8b0cb" Jul 10 16:58:52.769913 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.769895 2567 scope.go:117] "RemoveContainer" containerID="d1403391a4a5b1070c917358924723dc200b96f4093ac450eb117bcec5a8b0cb" Jul 10 16:58:52.770191 ip-10-0-131-207 kubenswrapper[2567]: E0710 16:58:52.770170 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1403391a4a5b1070c917358924723dc200b96f4093ac450eb117bcec5a8b0cb\": container with ID starting with d1403391a4a5b1070c917358924723dc200b96f4093ac450eb117bcec5a8b0cb not found: ID does not exist" containerID="d1403391a4a5b1070c917358924723dc200b96f4093ac450eb117bcec5a8b0cb" Jul 10 16:58:52.770264 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.770199 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1403391a4a5b1070c917358924723dc200b96f4093ac450eb117bcec5a8b0cb"} err="failed to get container status \"d1403391a4a5b1070c917358924723dc200b96f4093ac450eb117bcec5a8b0cb\": rpc error: code = NotFound desc = could not find container \"d1403391a4a5b1070c917358924723dc200b96f4093ac450eb117bcec5a8b0cb\": container with ID starting with d1403391a4a5b1070c917358924723dc200b96f4093ac450eb117bcec5a8b0cb not found: ID does not exist" Jul 10 16:58:52.777989 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.777956 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-794b7dd9d7-9bkbj"] Jul 10 16:58:52.779766 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:52.779746 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-794b7dd9d7-9bkbj"] Jul 10 16:58:54.680574 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:58:54.680531 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2f29d32-c014-427e-9275-788e2eb3d379" path="/var/lib/kubelet/pods/b2f29d32-c014-427e-9275-788e2eb3d379/volumes" Jul 10 16:59:30.505645 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:30.505619 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 16:59:30.507441 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:30.507421 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 16:59:30.512454 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:30.512434 2567 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 16:59:38.817957 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.817919 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz"] Jul 10 16:59:38.819425 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.818150 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f29d32-c014-427e-9275-788e2eb3d379" containerName="console" Jul 10 16:59:38.819425 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.818161 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f29d32-c014-427e-9275-788e2eb3d379" containerName="console" Jul 10 16:59:38.819425 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.818205 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f29d32-c014-427e-9275-788e2eb3d379" containerName="console" Jul 10 16:59:38.819842 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.819826 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz" Jul 10 16:59:38.823505 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.823480 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 10 16:59:38.823959 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.823936 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:59:38.824048 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.823992 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-tlzxt\"" Jul 10 16:59:38.854944 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.852843 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz"] Jul 10 16:59:38.883627 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.883594 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85fb3a5e-6fa7-4c6d-abde-6d585479372c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-fczkz\" (UID: \"85fb3a5e-6fa7-4c6d-abde-6d585479372c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz" Jul 10 16:59:38.883790 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.883641 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nscs\" (UniqueName: \"kubernetes.io/projected/85fb3a5e-6fa7-4c6d-abde-6d585479372c-kube-api-access-6nscs\") pod \"cert-manager-operator-controller-manager-55c444bf86-fczkz\" (UID: \"85fb3a5e-6fa7-4c6d-abde-6d585479372c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz" Jul 10 16:59:38.984581 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.984536 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85fb3a5e-6fa7-4c6d-abde-6d585479372c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-fczkz\" (UID: \"85fb3a5e-6fa7-4c6d-abde-6d585479372c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz" Jul 10 16:59:38.984753 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.984590 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nscs\" (UniqueName: \"kubernetes.io/projected/85fb3a5e-6fa7-4c6d-abde-6d585479372c-kube-api-access-6nscs\") pod \"cert-manager-operator-controller-manager-55c444bf86-fczkz\" (UID: \"85fb3a5e-6fa7-4c6d-abde-6d585479372c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz" Jul 10 16:59:38.984922 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.984902 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85fb3a5e-6fa7-4c6d-abde-6d585479372c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-fczkz\" (UID: \"85fb3a5e-6fa7-4c6d-abde-6d585479372c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz" Jul 10 16:59:38.998467 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:38.998435 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nscs\" (UniqueName: \"kubernetes.io/projected/85fb3a5e-6fa7-4c6d-abde-6d585479372c-kube-api-access-6nscs\") pod \"cert-manager-operator-controller-manager-55c444bf86-fczkz\" (UID: \"85fb3a5e-6fa7-4c6d-abde-6d585479372c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz" Jul 10 16:59:39.128936 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:39.128857 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz" Jul 10 16:59:39.259480 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:39.259449 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz"] Jul 10 16:59:39.265096 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:59:39.265067 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85fb3a5e_6fa7_4c6d_abde_6d585479372c.slice/crio-0f0495cd385d01167a7535dcb9108e62216832eda99446f1ba85d9b9b7b2e8fe WatchSource:0}: Error finding container 0f0495cd385d01167a7535dcb9108e62216832eda99446f1ba85d9b9b7b2e8fe: Status 404 returned error can't find the container with id 0f0495cd385d01167a7535dcb9108e62216832eda99446f1ba85d9b9b7b2e8fe Jul 10 16:59:39.266874 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:39.266857 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:59:39.916351 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:39.916312 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz" event={"ID":"85fb3a5e-6fa7-4c6d-abde-6d585479372c","Type":"ContainerStarted","Data":"0f0495cd385d01167a7535dcb9108e62216832eda99446f1ba85d9b9b7b2e8fe"} Jul 10 16:59:41.926289 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:41.926192 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz" event={"ID":"85fb3a5e-6fa7-4c6d-abde-6d585479372c","Type":"ContainerStarted","Data":"6089537a574d308f33446a2720009afb852db8f5f91d210a7cd5457e94014c56"} Jul 10 16:59:41.947894 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:41.947607 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fczkz" podStartSLOduration=1.591241066 podStartE2EDuration="3.947591313s" totalImagesPullingTime="2.356350247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:59:41.947560647 +0000 UTC m=+311.834852001" watchObservedRunningTime="2026-07-10 16:59:41.947591313 +0000 UTC m=+311.834882613" Jul 10 16:59:50.434477 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.434437 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5"] Jul 10 16:59:50.436568 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.436550 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5" Jul 10 16:59:50.439093 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.439067 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 16:59:50.440228 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.440206 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 16:59:50.440396 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.440301 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8v745\"" Jul 10 16:59:50.444331 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.444309 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5"] Jul 10 16:59:50.463156 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.463127 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr9xb\" (UniqueName: \"kubernetes.io/projected/72bd1dc0-b406-4806-84d4-f04cec691c95-kube-api-access-qr9xb\") pod \"cert-manager-cainjector-559cdd8c87-jf7x5\" (UID: \"72bd1dc0-b406-4806-84d4-f04cec691c95\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5" Jul 10 16:59:50.463315 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.463285 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72bd1dc0-b406-4806-84d4-f04cec691c95-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jf7x5\" (UID: \"72bd1dc0-b406-4806-84d4-f04cec691c95\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5" Jul 10 16:59:50.563826 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.563789 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72bd1dc0-b406-4806-84d4-f04cec691c95-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jf7x5\" (UID: \"72bd1dc0-b406-4806-84d4-f04cec691c95\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5" Jul 10 16:59:50.563980 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.563838 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr9xb\" (UniqueName: \"kubernetes.io/projected/72bd1dc0-b406-4806-84d4-f04cec691c95-kube-api-access-qr9xb\") pod \"cert-manager-cainjector-559cdd8c87-jf7x5\" (UID: \"72bd1dc0-b406-4806-84d4-f04cec691c95\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5" Jul 10 16:59:50.572445 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.572413 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72bd1dc0-b406-4806-84d4-f04cec691c95-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jf7x5\" (UID: \"72bd1dc0-b406-4806-84d4-f04cec691c95\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5" Jul 10 16:59:50.572550 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.572512 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr9xb\" (UniqueName: \"kubernetes.io/projected/72bd1dc0-b406-4806-84d4-f04cec691c95-kube-api-access-qr9xb\") pod \"cert-manager-cainjector-559cdd8c87-jf7x5\" (UID: \"72bd1dc0-b406-4806-84d4-f04cec691c95\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5" Jul 10 16:59:50.744967 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.744942 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5" Jul 10 16:59:50.871159 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.871115 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5"] Jul 10 16:59:50.873351 ip-10-0-131-207 kubenswrapper[2567]: W0710 16:59:50.873318 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72bd1dc0_b406_4806_84d4_f04cec691c95.slice/crio-06a09733e80d8907a2343bf0bd0af66748391ff326cf7cdb6c9fa15effff0d56 WatchSource:0}: Error finding container 06a09733e80d8907a2343bf0bd0af66748391ff326cf7cdb6c9fa15effff0d56: Status 404 returned error can't find the container with id 06a09733e80d8907a2343bf0bd0af66748391ff326cf7cdb6c9fa15effff0d56 Jul 10 16:59:50.956701 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:50.956666 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5" event={"ID":"72bd1dc0-b406-4806-84d4-f04cec691c95","Type":"ContainerStarted","Data":"06a09733e80d8907a2343bf0bd0af66748391ff326cf7cdb6c9fa15effff0d56"} Jul 10 16:59:55.977993 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:55.977954 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5" event={"ID":"72bd1dc0-b406-4806-84d4-f04cec691c95","Type":"ContainerStarted","Data":"3708a1501352dc3d41e6a0c948435f124f277e6baa99f441f621b2d7d820d88b"} Jul 10 16:59:56.002406 ip-10-0-131-207 kubenswrapper[2567]: I0710 16:59:56.002350 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jf7x5" podStartSLOduration=1.512371952 podStartE2EDuration="6.002331793s" totalImagesPullingTime="4.489959841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:59:56.001448761 +0000 UTC m=+325.888740080" watchObservedRunningTime="2026-07-10 16:59:56.002331793 +0000 UTC m=+325.889623113" Jul 10 17:00:20.112237 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.112206 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6"] Jul 10 17:00:20.116986 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.116967 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.120613 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.120592 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 17:00:20.120922 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.120902 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 17:00:20.121204 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.121189 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gvgdz\"" Jul 10 17:00:20.121253 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.121200 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 17:00:20.121253 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.121200 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 17:00:20.130867 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.130842 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6"] Jul 10 17:00:20.182225 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.182191 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.182418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.182255 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srsj8\" (UniqueName: \"kubernetes.io/projected/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-kube-api-access-srsj8\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.182418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.182300 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-manifests\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.182418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.182330 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-tmp\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.182418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.182350 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-webhook-cert\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.283074 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.283042 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srsj8\" (UniqueName: \"kubernetes.io/projected/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-kube-api-access-srsj8\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.283254 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.283080 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-manifests\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.283356 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.283258 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-tmp\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.283356 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.283313 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-webhook-cert\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.283356 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.283348 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.283514 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.283479 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-manifests\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.285783 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.285749 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-tmp\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.285961 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.285945 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.286027 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.285979 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-webhook-cert\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.292777 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.292754 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srsj8\" (UniqueName: \"kubernetes.io/projected/007e50fc-7ce7-4946-89b9-9dcbb3d0c18a-kube-api-access-srsj8\") pod \"opendatahub-operator-controller-manager-67764b8b78-jdvj6\" (UID: \"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.427418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.427318 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:20.565534 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:20.565508 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6"] Jul 10 17:00:20.568395 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:00:20.568370 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007e50fc_7ce7_4946_89b9_9dcbb3d0c18a.slice/crio-3e92b1ef8db01cdb4dba03069851dc76af147e36a0197de9e7f0713fae1a3786 WatchSource:0}: Error finding container 3e92b1ef8db01cdb4dba03069851dc76af147e36a0197de9e7f0713fae1a3786: Status 404 returned error can't find the container with id 3e92b1ef8db01cdb4dba03069851dc76af147e36a0197de9e7f0713fae1a3786 Jul 10 17:00:21.066244 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:21.066204 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" event={"ID":"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a","Type":"ContainerStarted","Data":"3e92b1ef8db01cdb4dba03069851dc76af147e36a0197de9e7f0713fae1a3786"} Jul 10 17:00:24.081064 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:24.080978 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" event={"ID":"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a","Type":"ContainerStarted","Data":"c66a46762ec1ca91ef5cd0e48ec81c2ff7d97407754470c7670dd41fc1e4e23a"} Jul 10 17:00:25.086707 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:25.086672 2567 generic.go:356] "Generic (PLEG): container finished" podID="007e50fc-7ce7-4946-89b9-9dcbb3d0c18a" containerID="c66a46762ec1ca91ef5cd0e48ec81c2ff7d97407754470c7670dd41fc1e4e23a" exitCode=0 Jul 10 17:00:25.087077 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:25.086741 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" event={"ID":"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a","Type":"ContainerDied","Data":"c66a46762ec1ca91ef5cd0e48ec81c2ff7d97407754470c7670dd41fc1e4e23a"} Jul 10 17:00:26.093085 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:26.093047 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" event={"ID":"007e50fc-7ce7-4946-89b9-9dcbb3d0c18a","Type":"ContainerStarted","Data":"0d858174ff9a7ae26901549f973aa23d21d565fbc4d6422880000e77176e5886"} Jul 10 17:00:26.093618 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:26.093199 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:26.125222 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:26.125169 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" podStartSLOduration=1.315471026 podStartE2EDuration="6.125151183s" totalImagesPullingTime="3.801433781s" totalInitContainerRuntime="1.008246376s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:00:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:00:26.123992919 +0000 UTC m=+356.011284237" watchObservedRunningTime="2026-07-10 17:00:26.125151183 +0000 UTC m=+356.012442503" Jul 10 17:00:37.098974 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:37.098942 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-jdvj6" Jul 10 17:00:40.672416 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.672377 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-558b5454d6-85xfq"] Jul 10 17:00:40.684027 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.683986 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" Jul 10 17:00:40.687351 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.687320 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 17:00:40.688111 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.688087 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 17:00:40.688261 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.688117 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mck87\"" Jul 10 17:00:40.688261 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.688189 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 17:00:40.688499 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.688481 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 17:00:40.704390 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.704358 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-558b5454d6-85xfq"] Jul 10 17:00:40.847212 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.847166 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5a1938-c8af-431e-be5c-6d4bb9f4a171-tls-certs\") pod \"kube-auth-proxy-558b5454d6-85xfq\" (UID: \"3b5a1938-c8af-431e-be5c-6d4bb9f4a171\") " pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" Jul 10 17:00:40.847435 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.847243 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gx5x\" (UniqueName: \"kubernetes.io/projected/3b5a1938-c8af-431e-be5c-6d4bb9f4a171-kube-api-access-8gx5x\") pod \"kube-auth-proxy-558b5454d6-85xfq\" (UID: \"3b5a1938-c8af-431e-be5c-6d4bb9f4a171\") " pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" Jul 10 17:00:40.847435 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.847301 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b5a1938-c8af-431e-be5c-6d4bb9f4a171-tmp\") pod \"kube-auth-proxy-558b5454d6-85xfq\" (UID: \"3b5a1938-c8af-431e-be5c-6d4bb9f4a171\") " pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" Jul 10 17:00:40.948470 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.948430 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5a1938-c8af-431e-be5c-6d4bb9f4a171-tls-certs\") pod \"kube-auth-proxy-558b5454d6-85xfq\" (UID: \"3b5a1938-c8af-431e-be5c-6d4bb9f4a171\") " pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" Jul 10 17:00:40.948631 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.948497 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gx5x\" (UniqueName: \"kubernetes.io/projected/3b5a1938-c8af-431e-be5c-6d4bb9f4a171-kube-api-access-8gx5x\") pod \"kube-auth-proxy-558b5454d6-85xfq\" (UID: \"3b5a1938-c8af-431e-be5c-6d4bb9f4a171\") " pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" Jul 10 17:00:40.948631 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.948524 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b5a1938-c8af-431e-be5c-6d4bb9f4a171-tmp\") pod \"kube-auth-proxy-558b5454d6-85xfq\" (UID: \"3b5a1938-c8af-431e-be5c-6d4bb9f4a171\") " pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" Jul 10 17:00:40.950944 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.950912 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b5a1938-c8af-431e-be5c-6d4bb9f4a171-tmp\") pod \"kube-auth-proxy-558b5454d6-85xfq\" (UID: \"3b5a1938-c8af-431e-be5c-6d4bb9f4a171\") " pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" Jul 10 17:00:40.951245 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.951223 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5a1938-c8af-431e-be5c-6d4bb9f4a171-tls-certs\") pod \"kube-auth-proxy-558b5454d6-85xfq\" (UID: \"3b5a1938-c8af-431e-be5c-6d4bb9f4a171\") " pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" Jul 10 17:00:40.958753 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.958725 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gx5x\" (UniqueName: \"kubernetes.io/projected/3b5a1938-c8af-431e-be5c-6d4bb9f4a171-kube-api-access-8gx5x\") pod \"kube-auth-proxy-558b5454d6-85xfq\" (UID: \"3b5a1938-c8af-431e-be5c-6d4bb9f4a171\") " pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" Jul 10 17:00:40.964435 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.964409 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-l2csp"] Jul 10 17:00:40.967780 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.967763 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" Jul 10 17:00:40.970379 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.970360 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 17:00:40.970491 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.970385 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jnhg8\"" Jul 10 17:00:40.975834 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.975808 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-l2csp"] Jul 10 17:00:40.993917 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:40.993883 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" Jul 10 17:00:41.049833 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:41.049694 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2276b013-5fbb-4763-b462-1cd63e49e432-cert\") pod \"odh-model-controller-7998c67dc4-l2csp\" (UID: \"2276b013-5fbb-4763-b462-1cd63e49e432\") " pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" Jul 10 17:00:41.049833 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:41.049759 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p8th\" (UniqueName: \"kubernetes.io/projected/2276b013-5fbb-4763-b462-1cd63e49e432-kube-api-access-8p8th\") pod \"odh-model-controller-7998c67dc4-l2csp\" (UID: \"2276b013-5fbb-4763-b462-1cd63e49e432\") " pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" Jul 10 17:00:41.141929 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:41.141897 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-558b5454d6-85xfq"] Jul 10 17:00:41.145061 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:00:41.145021 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5a1938_c8af_431e_be5c_6d4bb9f4a171.slice/crio-33d67f53ff32e926ba6b79a82067abe1f8a67ff4546435eb38dc77ce1aed707d WatchSource:0}: Error finding container 33d67f53ff32e926ba6b79a82067abe1f8a67ff4546435eb38dc77ce1aed707d: Status 404 returned error can't find the container with id 33d67f53ff32e926ba6b79a82067abe1f8a67ff4546435eb38dc77ce1aed707d Jul 10 17:00:41.150140 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:41.150114 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2276b013-5fbb-4763-b462-1cd63e49e432-cert\") pod \"odh-model-controller-7998c67dc4-l2csp\" (UID: \"2276b013-5fbb-4763-b462-1cd63e49e432\") " pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" Jul 10 17:00:41.150257 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:41.150170 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p8th\" (UniqueName: \"kubernetes.io/projected/2276b013-5fbb-4763-b462-1cd63e49e432-kube-api-access-8p8th\") pod \"odh-model-controller-7998c67dc4-l2csp\" (UID: \"2276b013-5fbb-4763-b462-1cd63e49e432\") " pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" Jul 10 17:00:41.150342 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:00:41.150252 2567 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 17:00:41.150396 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:00:41.150347 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2276b013-5fbb-4763-b462-1cd63e49e432-cert podName:2276b013-5fbb-4763-b462-1cd63e49e432 nodeName:}" failed. No retries permitted until 2026-07-10 17:00:41.650325156 +0000 UTC m=+371.537616467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2276b013-5fbb-4763-b462-1cd63e49e432-cert") pod "odh-model-controller-7998c67dc4-l2csp" (UID: "2276b013-5fbb-4763-b462-1cd63e49e432") : secret "odh-model-controller-webhook-cert" not found Jul 10 17:00:41.151135 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:41.151110 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" event={"ID":"3b5a1938-c8af-431e-be5c-6d4bb9f4a171","Type":"ContainerStarted","Data":"33d67f53ff32e926ba6b79a82067abe1f8a67ff4546435eb38dc77ce1aed707d"} Jul 10 17:00:41.158875 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:41.158850 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p8th\" (UniqueName: \"kubernetes.io/projected/2276b013-5fbb-4763-b462-1cd63e49e432-kube-api-access-8p8th\") pod \"odh-model-controller-7998c67dc4-l2csp\" (UID: \"2276b013-5fbb-4763-b462-1cd63e49e432\") " pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" Jul 10 17:00:41.654950 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:41.654909 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2276b013-5fbb-4763-b462-1cd63e49e432-cert\") pod \"odh-model-controller-7998c67dc4-l2csp\" (UID: \"2276b013-5fbb-4763-b462-1cd63e49e432\") " pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" Jul 10 17:00:41.657725 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:41.657698 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2276b013-5fbb-4763-b462-1cd63e49e432-cert\") pod \"odh-model-controller-7998c67dc4-l2csp\" (UID: \"2276b013-5fbb-4763-b462-1cd63e49e432\") " pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" Jul 10 17:00:41.878402 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:41.878354 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" Jul 10 17:00:42.035789 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.035755 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-l2csp"] Jul 10 17:00:42.038908 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:00:42.038884 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2276b013_5fbb_4763_b462_1cd63e49e432.slice/crio-5eecd35de6498f31e2d8815190dd01c9ba18f333375b562273ff7aa3fbfc6cab WatchSource:0}: Error finding container 5eecd35de6498f31e2d8815190dd01c9ba18f333375b562273ff7aa3fbfc6cab: Status 404 returned error can't find the container with id 5eecd35de6498f31e2d8815190dd01c9ba18f333375b562273ff7aa3fbfc6cab Jul 10 17:00:42.155204 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.155170 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-flc2x"] Jul 10 17:00:42.159266 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.159232 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" event={"ID":"2276b013-5fbb-4763-b462-1cd63e49e432","Type":"ContainerStarted","Data":"5eecd35de6498f31e2d8815190dd01c9ba18f333375b562273ff7aa3fbfc6cab"} Jul 10 17:00:42.159458 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.159379 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-flc2x" Jul 10 17:00:42.162334 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.162131 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 17:00:42.162334 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.162135 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-n5nqw\"" Jul 10 17:00:42.174666 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.174636 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-flc2x"] Jul 10 17:00:42.260449 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.260414 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13085247-748f-408f-b5b1-504973bfd245-tls-certs\") pod \"model-serving-api-54d89bd5fc-flc2x\" (UID: \"13085247-748f-408f-b5b1-504973bfd245\") " pod="opendatahub/model-serving-api-54d89bd5fc-flc2x" Jul 10 17:00:42.260449 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.260459 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc2f7\" (UniqueName: \"kubernetes.io/projected/13085247-748f-408f-b5b1-504973bfd245-kube-api-access-gc2f7\") pod \"model-serving-api-54d89bd5fc-flc2x\" (UID: \"13085247-748f-408f-b5b1-504973bfd245\") " pod="opendatahub/model-serving-api-54d89bd5fc-flc2x" Jul 10 17:00:42.308626 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.308594 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc"] Jul 10 17:00:42.312082 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.312054 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.315781 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.315754 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 17:00:42.315925 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.315754 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 17:00:42.315925 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.315754 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 17:00:42.316066 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.315767 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5fql7\"" Jul 10 17:00:42.316190 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.316108 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 17:00:42.331728 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.331677 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc"] Jul 10 17:00:42.360966 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.360926 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13085247-748f-408f-b5b1-504973bfd245-tls-certs\") pod \"model-serving-api-54d89bd5fc-flc2x\" (UID: \"13085247-748f-408f-b5b1-504973bfd245\") " pod="opendatahub/model-serving-api-54d89bd5fc-flc2x" Jul 10 17:00:42.360966 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.360977 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2f7\" (UniqueName: \"kubernetes.io/projected/13085247-748f-408f-b5b1-504973bfd245-kube-api-access-gc2f7\") pod \"model-serving-api-54d89bd5fc-flc2x\" (UID: \"13085247-748f-408f-b5b1-504973bfd245\") " pod="opendatahub/model-serving-api-54d89bd5fc-flc2x" Jul 10 17:00:42.364016 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.363986 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13085247-748f-408f-b5b1-504973bfd245-tls-certs\") pod \"model-serving-api-54d89bd5fc-flc2x\" (UID: \"13085247-748f-408f-b5b1-504973bfd245\") " pod="opendatahub/model-serving-api-54d89bd5fc-flc2x" Jul 10 17:00:42.370130 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.370099 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc2f7\" (UniqueName: \"kubernetes.io/projected/13085247-748f-408f-b5b1-504973bfd245-kube-api-access-gc2f7\") pod \"model-serving-api-54d89bd5fc-flc2x\" (UID: \"13085247-748f-408f-b5b1-504973bfd245\") " pod="opendatahub/model-serving-api-54d89bd5fc-flc2x" Jul 10 17:00:42.461814 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.461769 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.461814 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.461814 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.462058 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.461853 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/630ffad7-ca34-40a7-82e1-6781adfbab27-local-certs\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.462058 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.461870 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbv8d\" (UniqueName: \"kubernetes.io/projected/630ffad7-ca34-40a7-82e1-6781adfbab27-kube-api-access-pbv8d\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.462058 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.461899 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-cacerts\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.462058 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.461918 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.462058 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.462008 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-token\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.471803 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.471767 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-flc2x" Jul 10 17:00:42.562791 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.562752 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.562791 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.562792 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.563036 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.562832 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/630ffad7-ca34-40a7-82e1-6781adfbab27-local-certs\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.563036 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.562850 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbv8d\" (UniqueName: \"kubernetes.io/projected/630ffad7-ca34-40a7-82e1-6781adfbab27-kube-api-access-pbv8d\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.563036 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.562870 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-cacerts\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.563036 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.562899 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.563036 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.562947 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-token\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.563842 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.563490 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.566324 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.566296 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/630ffad7-ca34-40a7-82e1-6781adfbab27-local-certs\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.566451 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.566359 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.566518 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.566461 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-cacerts\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.567115 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.567089 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.572215 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.572172 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-token\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.572839 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.572699 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbv8d\" (UniqueName: \"kubernetes.io/projected/630ffad7-ca34-40a7-82e1-6781adfbab27-kube-api-access-pbv8d\") pod \"istiod-openshift-gateway-7574459656-dxtbc\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.607565 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.607523 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-flc2x"] Jul 10 17:00:42.610438 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:00:42.610412 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13085247_748f_408f_b5b1_504973bfd245.slice/crio-734cb66df73dc84ad8687d09bc9926b82d2bad0b8264652779d2127afca7845b WatchSource:0}: Error finding container 734cb66df73dc84ad8687d09bc9926b82d2bad0b8264652779d2127afca7845b: Status 404 returned error can't find the container with id 734cb66df73dc84ad8687d09bc9926b82d2bad0b8264652779d2127afca7845b Jul 10 17:00:42.623264 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.623222 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:42.769690 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:42.769667 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc"] Jul 10 17:00:42.771757 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:00:42.771693 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630ffad7_ca34_40a7_82e1_6781adfbab27.slice/crio-b938625b7959f7975f369efe212c188449cf628e88b610575d162084f28de300 WatchSource:0}: Error finding container b938625b7959f7975f369efe212c188449cf628e88b610575d162084f28de300: Status 404 returned error can't find the container with id b938625b7959f7975f369efe212c188449cf628e88b610575d162084f28de300 Jul 10 17:00:43.162597 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:43.162497 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" event={"ID":"630ffad7-ca34-40a7-82e1-6781adfbab27","Type":"ContainerStarted","Data":"b938625b7959f7975f369efe212c188449cf628e88b610575d162084f28de300"} Jul 10 17:00:43.163542 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:43.163516 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-flc2x" event={"ID":"13085247-748f-408f-b5b1-504973bfd245","Type":"ContainerStarted","Data":"734cb66df73dc84ad8687d09bc9926b82d2bad0b8264652779d2127afca7845b"} Jul 10 17:00:47.184478 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:47.184176 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-flc2x" event={"ID":"13085247-748f-408f-b5b1-504973bfd245","Type":"ContainerStarted","Data":"188a839eceffbf2164f9555bc78a45998f804eee539320efce39b4272b947a8d"} Jul 10 17:00:47.184478 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:47.184300 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-flc2x" Jul 10 17:00:47.186292 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:47.186249 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" event={"ID":"3b5a1938-c8af-431e-be5c-6d4bb9f4a171","Type":"ContainerStarted","Data":"c39d934f7baa5e9bf13e2c2faa1f6b3146e33213bd2bf39a44049289f98e3af2"} Jul 10 17:00:47.189006 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:47.188979 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" event={"ID":"2276b013-5fbb-4763-b462-1cd63e49e432","Type":"ContainerStarted","Data":"6347bad4176b573d6fa7ecd795f81dad8bfec319ef4ddea6b61e23951fbd3a18"} Jul 10 17:00:47.189185 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:47.189165 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" Jul 10 17:00:47.205648 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:47.205580 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-flc2x" podStartSLOduration=1.327302114 podStartE2EDuration="5.205564147s" totalImagesPullingTime="3.878262033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:00:47.205066952 +0000 UTC m=+377.092358272" watchObservedRunningTime="2026-07-10 17:00:47.205564147 +0000 UTC m=+377.092855500" Jul 10 17:00:47.225867 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:47.225806 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-558b5454d6-85xfq" podStartSLOduration=1.88264462 podStartE2EDuration="7.225788628s" totalImagesPullingTime="5.343144008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:00:47.22436362 +0000 UTC m=+377.111654963" watchObservedRunningTime="2026-07-10 17:00:47.225788628 +0000 UTC m=+377.113079948" Jul 10 17:00:47.247228 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:47.247168 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" podStartSLOduration=2.787399067 podStartE2EDuration="7.24714994s" totalImagesPullingTime="4.459750873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:00:47.245595298 +0000 UTC m=+377.132886640" watchObservedRunningTime="2026-07-10 17:00:47.24714994 +0000 UTC m=+377.134441260" Jul 10 17:00:47.991340 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:47.991304 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:00:47.991444 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:47.991387 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:00:48.195724 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:48.195673 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" event={"ID":"630ffad7-ca34-40a7-82e1-6781adfbab27","Type":"ContainerStarted","Data":"c0d997bb30352c5bb3de91e3f62ea462743e71538ea14cee83c351afd97953c8"} Jul 10 17:00:48.196174 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:48.195911 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:48.227372 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:48.227326 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" podStartSLOduration=1.010077284 podStartE2EDuration="6.227310416s" totalImagesPullingTime="5.217233132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:00:48.226055251 +0000 UTC m=+378.113346571" watchObservedRunningTime="2026-07-10 17:00:48.227310416 +0000 UTC m=+378.114601736" Jul 10 17:00:49.201530 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:49.201490 2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-dxtbc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 17:00:49.201872 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:49.201556 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" podUID="630ffad7-ca34-40a7-82e1-6781adfbab27" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:00:51.722956 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:51.722919 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-j98wg"] Jul 10 17:00:51.725825 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:51.725809 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" Jul 10 17:00:51.731807 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:51.731778 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wnqdx\"" Jul 10 17:00:51.732775 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:51.732751 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 17:00:51.750485 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:51.750452 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-j98wg"] Jul 10 17:00:51.850044 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:51.850004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12a523b0-50b9-4449-b3be-d66f9cf856de-cert\") pod \"kserve-controller-manager-d6f4b65b8-j98wg\" (UID: \"12a523b0-50b9-4449-b3be-d66f9cf856de\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" Jul 10 17:00:51.850044 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:51.850048 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hf2g\" (UniqueName: \"kubernetes.io/projected/12a523b0-50b9-4449-b3be-d66f9cf856de-kube-api-access-2hf2g\") pod \"kserve-controller-manager-d6f4b65b8-j98wg\" (UID: \"12a523b0-50b9-4449-b3be-d66f9cf856de\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" Jul 10 17:00:51.951247 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:51.951210 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12a523b0-50b9-4449-b3be-d66f9cf856de-cert\") pod \"kserve-controller-manager-d6f4b65b8-j98wg\" (UID: \"12a523b0-50b9-4449-b3be-d66f9cf856de\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" Jul 10 17:00:51.951247 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:51.951253 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hf2g\" (UniqueName: \"kubernetes.io/projected/12a523b0-50b9-4449-b3be-d66f9cf856de-kube-api-access-2hf2g\") pod \"kserve-controller-manager-d6f4b65b8-j98wg\" (UID: \"12a523b0-50b9-4449-b3be-d66f9cf856de\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" Jul 10 17:00:51.951497 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:00:51.951390 2567 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 10 17:00:51.951497 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:00:51.951472 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a523b0-50b9-4449-b3be-d66f9cf856de-cert podName:12a523b0-50b9-4449-b3be-d66f9cf856de nodeName:}" failed. No retries permitted until 2026-07-10 17:00:52.451451591 +0000 UTC m=+382.338742889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/12a523b0-50b9-4449-b3be-d66f9cf856de-cert") pod "kserve-controller-manager-d6f4b65b8-j98wg" (UID: "12a523b0-50b9-4449-b3be-d66f9cf856de") : secret "kserve-webhook-server-cert" not found Jul 10 17:00:51.967179 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:51.967143 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hf2g\" (UniqueName: \"kubernetes.io/projected/12a523b0-50b9-4449-b3be-d66f9cf856de-kube-api-access-2hf2g\") pod \"kserve-controller-manager-d6f4b65b8-j98wg\" (UID: \"12a523b0-50b9-4449-b3be-d66f9cf856de\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" Jul 10 17:00:52.201479 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:52.201439 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:52.455314 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:52.455189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12a523b0-50b9-4449-b3be-d66f9cf856de-cert\") pod \"kserve-controller-manager-d6f4b65b8-j98wg\" (UID: \"12a523b0-50b9-4449-b3be-d66f9cf856de\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" Jul 10 17:00:52.457818 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:52.457794 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12a523b0-50b9-4449-b3be-d66f9cf856de-cert\") pod \"kserve-controller-manager-d6f4b65b8-j98wg\" (UID: \"12a523b0-50b9-4449-b3be-d66f9cf856de\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" Jul 10 17:00:52.636362 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:52.636326 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" Jul 10 17:00:52.779057 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:52.779034 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-j98wg"] Jul 10 17:00:52.779907 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:00:52.779885 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a523b0_50b9_4449_b3be_d66f9cf856de.slice/crio-a2d495b7de7461d5ac01c98e0ee33c50390c684ee47ba16ddd7c54012b4a1835 WatchSource:0}: Error finding container a2d495b7de7461d5ac01c98e0ee33c50390c684ee47ba16ddd7c54012b4a1835: Status 404 returned error can't find the container with id a2d495b7de7461d5ac01c98e0ee33c50390c684ee47ba16ddd7c54012b4a1835 Jul 10 17:00:53.215383 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.215347 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" event={"ID":"12a523b0-50b9-4449-b3be-d66f9cf856de","Type":"ContainerStarted","Data":"a2d495b7de7461d5ac01c98e0ee33c50390c684ee47ba16ddd7c54012b4a1835"} Jul 10 17:00:53.514806 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.514717 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc"] Jul 10 17:00:53.514946 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.514890 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" podUID="630ffad7-ca34-40a7-82e1-6781adfbab27" containerName="discovery" containerID="cri-o://c0d997bb30352c5bb3de91e3f62ea462743e71538ea14cee83c351afd97953c8" gracePeriod=30 Jul 10 17:00:53.759154 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.759125 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:53.866244 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.866162 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/630ffad7-ca34-40a7-82e1-6781adfbab27-local-certs\") pod \"630ffad7-ca34-40a7-82e1-6781adfbab27\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " Jul 10 17:00:53.866712 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.866245 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-token\") pod \"630ffad7-ca34-40a7-82e1-6781adfbab27\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " Jul 10 17:00:53.866712 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.866303 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbv8d\" (UniqueName: \"kubernetes.io/projected/630ffad7-ca34-40a7-82e1-6781adfbab27-kube-api-access-pbv8d\") pod \"630ffad7-ca34-40a7-82e1-6781adfbab27\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " Jul 10 17:00:53.866712 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.866328 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-kubeconfig\") pod \"630ffad7-ca34-40a7-82e1-6781adfbab27\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " Jul 10 17:00:53.866712 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.866441 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-csr-ca-configmap\") pod \"630ffad7-ca34-40a7-82e1-6781adfbab27\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " Jul 10 17:00:53.866712 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.866503 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-csr-dns-cert\") pod \"630ffad7-ca34-40a7-82e1-6781adfbab27\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " Jul 10 17:00:53.866712 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.866596 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-cacerts\") pod \"630ffad7-ca34-40a7-82e1-6781adfbab27\" (UID: \"630ffad7-ca34-40a7-82e1-6781adfbab27\") " Jul 10 17:00:53.867457 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.867423 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "630ffad7-ca34-40a7-82e1-6781adfbab27" (UID: "630ffad7-ca34-40a7-82e1-6781adfbab27"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:00:53.869165 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.869112 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-token" (OuterVolumeSpecName: "istio-token") pod "630ffad7-ca34-40a7-82e1-6781adfbab27" (UID: "630ffad7-ca34-40a7-82e1-6781adfbab27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:00:53.869418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.869386 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "630ffad7-ca34-40a7-82e1-6781adfbab27" (UID: "630ffad7-ca34-40a7-82e1-6781adfbab27"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:00:53.869518 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.869467 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630ffad7-ca34-40a7-82e1-6781adfbab27-local-certs" (OuterVolumeSpecName: "local-certs") pod "630ffad7-ca34-40a7-82e1-6781adfbab27" (UID: "630ffad7-ca34-40a7-82e1-6781adfbab27"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:00:53.869586 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.869534 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630ffad7-ca34-40a7-82e1-6781adfbab27-kube-api-access-pbv8d" (OuterVolumeSpecName: "kube-api-access-pbv8d") pod "630ffad7-ca34-40a7-82e1-6781adfbab27" (UID: "630ffad7-ca34-40a7-82e1-6781adfbab27"). InnerVolumeSpecName "kube-api-access-pbv8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:00:53.869789 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.869767 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "630ffad7-ca34-40a7-82e1-6781adfbab27" (UID: "630ffad7-ca34-40a7-82e1-6781adfbab27"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:00:53.869900 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.869789 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-cacerts" (OuterVolumeSpecName: "cacerts") pod "630ffad7-ca34-40a7-82e1-6781adfbab27" (UID: "630ffad7-ca34-40a7-82e1-6781adfbab27"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:00:53.967725 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.967686 2567 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-cacerts\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:00:53.967725 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.967726 2567 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/630ffad7-ca34-40a7-82e1-6781adfbab27-local-certs\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:00:53.967977 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.967746 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-token\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:00:53.967977 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.967761 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbv8d\" (UniqueName: \"kubernetes.io/projected/630ffad7-ca34-40a7-82e1-6781adfbab27-kube-api-access-pbv8d\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:00:53.967977 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.967780 2567 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-kubeconfig\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:00:53.967977 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.967794 2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-csr-ca-configmap\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:00:53.967977 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:53.967828 2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/630ffad7-ca34-40a7-82e1-6781adfbab27-istio-csr-dns-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:00:54.221568 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:54.221525 2567 generic.go:356] "Generic (PLEG): container finished" podID="630ffad7-ca34-40a7-82e1-6781adfbab27" containerID="c0d997bb30352c5bb3de91e3f62ea462743e71538ea14cee83c351afd97953c8" exitCode=0 Jul 10 17:00:54.221757 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:54.221585 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" Jul 10 17:00:54.221757 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:54.221607 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" event={"ID":"630ffad7-ca34-40a7-82e1-6781adfbab27","Type":"ContainerDied","Data":"c0d997bb30352c5bb3de91e3f62ea462743e71538ea14cee83c351afd97953c8"} Jul 10 17:00:54.221757 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:54.221655 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc" event={"ID":"630ffad7-ca34-40a7-82e1-6781adfbab27","Type":"ContainerDied","Data":"b938625b7959f7975f369efe212c188449cf628e88b610575d162084f28de300"} Jul 10 17:00:54.221757 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:54.221674 2567 scope.go:117] "RemoveContainer" containerID="c0d997bb30352c5bb3de91e3f62ea462743e71538ea14cee83c351afd97953c8" Jul 10 17:00:54.234654 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:54.234628 2567 scope.go:117] "RemoveContainer" containerID="c0d997bb30352c5bb3de91e3f62ea462743e71538ea14cee83c351afd97953c8" Jul 10 17:00:54.235066 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:00:54.235036 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d997bb30352c5bb3de91e3f62ea462743e71538ea14cee83c351afd97953c8\": container with ID starting with c0d997bb30352c5bb3de91e3f62ea462743e71538ea14cee83c351afd97953c8 not found: ID does not exist" containerID="c0d997bb30352c5bb3de91e3f62ea462743e71538ea14cee83c351afd97953c8" Jul 10 17:00:54.235157 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:54.235079 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d997bb30352c5bb3de91e3f62ea462743e71538ea14cee83c351afd97953c8"} err="failed to get container status \"c0d997bb30352c5bb3de91e3f62ea462743e71538ea14cee83c351afd97953c8\": rpc error: code = NotFound desc = could not find container \"c0d997bb30352c5bb3de91e3f62ea462743e71538ea14cee83c351afd97953c8\": container with ID starting with c0d997bb30352c5bb3de91e3f62ea462743e71538ea14cee83c351afd97953c8 not found: ID does not exist" Jul 10 17:00:54.268832 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:54.268796 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc"] Jul 10 17:00:54.274151 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:54.274126 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dxtbc"] Jul 10 17:00:54.681964 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:54.681893 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="630ffad7-ca34-40a7-82e1-6781adfbab27" path="/var/lib/kubelet/pods/630ffad7-ca34-40a7-82e1-6781adfbab27/volumes" Jul 10 17:00:56.233913 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:56.233814 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" event={"ID":"12a523b0-50b9-4449-b3be-d66f9cf856de","Type":"ContainerStarted","Data":"bb8edc7fe9357d3dedfa159afed4dbf1cfff4f77fd55c70a1fd1332d4fd92f97"} Jul 10 17:00:56.234381 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:56.233918 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" Jul 10 17:00:56.252527 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:56.252479 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" podStartSLOduration=2.091544865 podStartE2EDuration="5.252465396s" totalImagesPullingTime="3.160920531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:00:56.251365977 +0000 UTC m=+386.138657310" watchObservedRunningTime="2026-07-10 17:00:56.252465396 +0000 UTC m=+386.139756715" Jul 10 17:00:58.198725 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:58.198691 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-l2csp" Jul 10 17:00:58.200579 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:00:58.200557 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-flc2x" Jul 10 17:01:27.243490 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:27.243460 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j98wg" Jul 10 17:01:41.716402 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:41.716371 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7ptzp"] Jul 10 17:01:41.717097 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:41.716642 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="630ffad7-ca34-40a7-82e1-6781adfbab27" containerName="discovery" Jul 10 17:01:41.717097 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:41.716652 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="630ffad7-ca34-40a7-82e1-6781adfbab27" containerName="discovery" Jul 10 17:01:41.717097 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:41.716705 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="630ffad7-ca34-40a7-82e1-6781adfbab27" containerName="discovery" Jul 10 17:01:41.719685 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:41.719669 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7ptzp" Jul 10 17:01:41.725154 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:41.725130 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 17:01:41.726021 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:41.726001 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-r6jzm\"" Jul 10 17:01:41.726021 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:41.726015 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 17:01:41.735229 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:41.735201 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7ptzp"] Jul 10 17:01:41.843466 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:41.843430 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh2vb\" (UniqueName: \"kubernetes.io/projected/a5574185-c533-4d18-92c9-a71b85985f4e-kube-api-access-xh2vb\") pod \"kuadrant-operator-catalog-7ptzp\" (UID: \"a5574185-c533-4d18-92c9-a71b85985f4e\") " pod="kuadrant-system/kuadrant-operator-catalog-7ptzp" Jul 10 17:01:41.944760 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:41.944718 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2vb\" (UniqueName: \"kubernetes.io/projected/a5574185-c533-4d18-92c9-a71b85985f4e-kube-api-access-xh2vb\") pod \"kuadrant-operator-catalog-7ptzp\" (UID: \"a5574185-c533-4d18-92c9-a71b85985f4e\") " pod="kuadrant-system/kuadrant-operator-catalog-7ptzp" Jul 10 17:01:41.953099 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:41.953065 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh2vb\" (UniqueName: \"kubernetes.io/projected/a5574185-c533-4d18-92c9-a71b85985f4e-kube-api-access-xh2vb\") pod \"kuadrant-operator-catalog-7ptzp\" (UID: \"a5574185-c533-4d18-92c9-a71b85985f4e\") " pod="kuadrant-system/kuadrant-operator-catalog-7ptzp" Jul 10 17:01:42.030080 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:42.029978 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7ptzp" Jul 10 17:01:42.073913 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:42.073877 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7ptzp"] Jul 10 17:01:42.154750 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:42.154728 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7ptzp"] Jul 10 17:01:42.156793 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:01:42.156768 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5574185_c533_4d18_92c9_a71b85985f4e.slice/crio-01d727d2d1e6b5d5222f050ddc088a8eb7c680d3dad61c9b111e22f033475c32 WatchSource:0}: Error finding container 01d727d2d1e6b5d5222f050ddc088a8eb7c680d3dad61c9b111e22f033475c32: Status 404 returned error can't find the container with id 01d727d2d1e6b5d5222f050ddc088a8eb7c680d3dad61c9b111e22f033475c32 Jul 10 17:01:42.281574 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:42.281495 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k7dz7"] Jul 10 17:01:42.284536 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:42.284519 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k7dz7" Jul 10 17:01:42.293194 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:42.293168 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k7dz7"] Jul 10 17:01:42.346786 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:42.346752 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf48c\" (UniqueName: \"kubernetes.io/projected/e193913a-dd57-464b-9e4d-b5d3e22af580-kube-api-access-rf48c\") pod \"kuadrant-operator-catalog-k7dz7\" (UID: \"e193913a-dd57-464b-9e4d-b5d3e22af580\") " pod="kuadrant-system/kuadrant-operator-catalog-k7dz7" Jul 10 17:01:42.420957 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:42.420920 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7ptzp" event={"ID":"a5574185-c533-4d18-92c9-a71b85985f4e","Type":"ContainerStarted","Data":"01d727d2d1e6b5d5222f050ddc088a8eb7c680d3dad61c9b111e22f033475c32"} Jul 10 17:01:42.447436 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:42.447400 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf48c\" (UniqueName: \"kubernetes.io/projected/e193913a-dd57-464b-9e4d-b5d3e22af580-kube-api-access-rf48c\") pod \"kuadrant-operator-catalog-k7dz7\" (UID: \"e193913a-dd57-464b-9e4d-b5d3e22af580\") " pod="kuadrant-system/kuadrant-operator-catalog-k7dz7" Jul 10 17:01:42.456316 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:42.456290 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf48c\" (UniqueName: \"kubernetes.io/projected/e193913a-dd57-464b-9e4d-b5d3e22af580-kube-api-access-rf48c\") pod \"kuadrant-operator-catalog-k7dz7\" (UID: \"e193913a-dd57-464b-9e4d-b5d3e22af580\") " pod="kuadrant-system/kuadrant-operator-catalog-k7dz7" Jul 10 17:01:42.595090 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:42.595001 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k7dz7" Jul 10 17:01:42.731732 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:42.731708 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k7dz7"] Jul 10 17:01:42.734133 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:01:42.734102 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode193913a_dd57_464b_9e4d_b5d3e22af580.slice/crio-e3b6f59bcae94decd4992320baaa733ddc46d749e46b058e51c89818b3d3c595 WatchSource:0}: Error finding container e3b6f59bcae94decd4992320baaa733ddc46d749e46b058e51c89818b3d3c595: Status 404 returned error can't find the container with id e3b6f59bcae94decd4992320baaa733ddc46d749e46b058e51c89818b3d3c595 Jul 10 17:01:43.427005 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:43.426964 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k7dz7" event={"ID":"e193913a-dd57-464b-9e4d-b5d3e22af580","Type":"ContainerStarted","Data":"e3b6f59bcae94decd4992320baaa733ddc46d749e46b058e51c89818b3d3c595"} Jul 10 17:01:45.436768 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:45.436735 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7ptzp" event={"ID":"a5574185-c533-4d18-92c9-a71b85985f4e","Type":"ContainerStarted","Data":"9fd27f1e6295fb78b5c3d0f179c716465ee573c321b07ac7b1735fc8f239446c"} Jul 10 17:01:45.437288 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:45.436822 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-7ptzp" podUID="a5574185-c533-4d18-92c9-a71b85985f4e" containerName="registry-server" containerID="cri-o://9fd27f1e6295fb78b5c3d0f179c716465ee573c321b07ac7b1735fc8f239446c" gracePeriod=2 Jul 10 17:01:45.438193 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:45.438167 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k7dz7" event={"ID":"e193913a-dd57-464b-9e4d-b5d3e22af580","Type":"ContainerStarted","Data":"50865349b7d2e60ad61a18d121894036f24c9d5ac738570c8eb88a68c58fc93c"} Jul 10 17:01:45.452227 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:45.452175 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7ptzp" podStartSLOduration=2.224889916 podStartE2EDuration="4.452159399s" totalImagesPullingTime="2.227269483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:01:45.452096258 +0000 UTC m=+435.339387597" watchObservedRunningTime="2026-07-10 17:01:45.452159399 +0000 UTC m=+435.339450720" Jul 10 17:01:45.467606 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:45.467552 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-k7dz7" podStartSLOduration=1.793980646 podStartE2EDuration="3.467536748s" totalImagesPullingTime="1.673556102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:01:45.466513408 +0000 UTC m=+435.353804728" watchObservedRunningTime="2026-07-10 17:01:45.467536748 +0000 UTC m=+435.354828068" Jul 10 17:01:45.675412 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:45.675388 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7ptzp" Jul 10 17:01:45.779079 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:45.779040 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh2vb\" (UniqueName: \"kubernetes.io/projected/a5574185-c533-4d18-92c9-a71b85985f4e-kube-api-access-xh2vb\") pod \"a5574185-c533-4d18-92c9-a71b85985f4e\" (UID: \"a5574185-c533-4d18-92c9-a71b85985f4e\") " Jul 10 17:01:45.781360 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:45.781332 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5574185-c533-4d18-92c9-a71b85985f4e-kube-api-access-xh2vb" (OuterVolumeSpecName: "kube-api-access-xh2vb") pod "a5574185-c533-4d18-92c9-a71b85985f4e" (UID: "a5574185-c533-4d18-92c9-a71b85985f4e"). InnerVolumeSpecName "kube-api-access-xh2vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:01:45.880093 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:45.880058 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh2vb\" (UniqueName: \"kubernetes.io/projected/a5574185-c533-4d18-92c9-a71b85985f4e-kube-api-access-xh2vb\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:01:46.442830 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:46.442791 2567 generic.go:356] "Generic (PLEG): container finished" podID="a5574185-c533-4d18-92c9-a71b85985f4e" containerID="9fd27f1e6295fb78b5c3d0f179c716465ee573c321b07ac7b1735fc8f239446c" exitCode=0 Jul 10 17:01:46.443213 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:46.442837 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7ptzp" Jul 10 17:01:46.443213 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:46.442883 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7ptzp" event={"ID":"a5574185-c533-4d18-92c9-a71b85985f4e","Type":"ContainerDied","Data":"9fd27f1e6295fb78b5c3d0f179c716465ee573c321b07ac7b1735fc8f239446c"} Jul 10 17:01:46.443213 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:46.442930 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7ptzp" event={"ID":"a5574185-c533-4d18-92c9-a71b85985f4e","Type":"ContainerDied","Data":"01d727d2d1e6b5d5222f050ddc088a8eb7c680d3dad61c9b111e22f033475c32"} Jul 10 17:01:46.443213 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:46.442950 2567 scope.go:117] "RemoveContainer" containerID="9fd27f1e6295fb78b5c3d0f179c716465ee573c321b07ac7b1735fc8f239446c" Jul 10 17:01:46.451745 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:46.451730 2567 scope.go:117] "RemoveContainer" containerID="9fd27f1e6295fb78b5c3d0f179c716465ee573c321b07ac7b1735fc8f239446c" Jul 10 17:01:46.452016 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:01:46.451991 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd27f1e6295fb78b5c3d0f179c716465ee573c321b07ac7b1735fc8f239446c\": container with ID starting with 9fd27f1e6295fb78b5c3d0f179c716465ee573c321b07ac7b1735fc8f239446c not found: ID does not exist" containerID="9fd27f1e6295fb78b5c3d0f179c716465ee573c321b07ac7b1735fc8f239446c" Jul 10 17:01:46.452081 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:46.452027 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd27f1e6295fb78b5c3d0f179c716465ee573c321b07ac7b1735fc8f239446c"} err="failed to get container status \"9fd27f1e6295fb78b5c3d0f179c716465ee573c321b07ac7b1735fc8f239446c\": rpc error: code = NotFound desc = could not find container \"9fd27f1e6295fb78b5c3d0f179c716465ee573c321b07ac7b1735fc8f239446c\": container with ID starting with 9fd27f1e6295fb78b5c3d0f179c716465ee573c321b07ac7b1735fc8f239446c not found: ID does not exist" Jul 10 17:01:46.463553 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:46.463531 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7ptzp"] Jul 10 17:01:46.467672 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:46.467651 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7ptzp"] Jul 10 17:01:46.681318 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:46.681266 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5574185-c533-4d18-92c9-a71b85985f4e" path="/var/lib/kubelet/pods/a5574185-c533-4d18-92c9-a71b85985f4e/volumes" Jul 10 17:01:52.595480 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:52.595391 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-k7dz7" Jul 10 17:01:52.595480 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:52.595456 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-k7dz7" Jul 10 17:01:52.618105 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:52.618081 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-k7dz7" Jul 10 17:01:53.494467 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:01:53.494432 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-k7dz7" Jul 10 17:02:11.220492 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.220455 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f6d755568-8gvcf"] Jul 10 17:02:11.220900 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.220742 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5574185-c533-4d18-92c9-a71b85985f4e" containerName="registry-server" Jul 10 17:02:11.220900 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.220754 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5574185-c533-4d18-92c9-a71b85985f4e" containerName="registry-server" Jul 10 17:02:11.220900 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.220801 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5574185-c533-4d18-92c9-a71b85985f4e" containerName="registry-server" Jul 10 17:02:11.223448 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.223428 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.241118 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.241084 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f6d755568-8gvcf"] Jul 10 17:02:11.379942 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.379902 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/427e095c-ad94-495d-b6d3-aa6ea806257a-service-ca\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.379942 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.379945 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/427e095c-ad94-495d-b6d3-aa6ea806257a-oauth-serving-cert\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.380151 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.379967 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/427e095c-ad94-495d-b6d3-aa6ea806257a-console-serving-cert\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.380151 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.380043 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/427e095c-ad94-495d-b6d3-aa6ea806257a-console-config\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.380151 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.380106 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/427e095c-ad94-495d-b6d3-aa6ea806257a-trusted-ca-bundle\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.380151 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.380129 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkhc6\" (UniqueName: \"kubernetes.io/projected/427e095c-ad94-495d-b6d3-aa6ea806257a-kube-api-access-tkhc6\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.380297 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.380175 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/427e095c-ad94-495d-b6d3-aa6ea806257a-console-oauth-config\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.480874 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.480767 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/427e095c-ad94-495d-b6d3-aa6ea806257a-trusted-ca-bundle\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.480874 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.480815 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkhc6\" (UniqueName: \"kubernetes.io/projected/427e095c-ad94-495d-b6d3-aa6ea806257a-kube-api-access-tkhc6\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.480874 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.480851 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/427e095c-ad94-495d-b6d3-aa6ea806257a-console-oauth-config\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.481131 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.480892 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/427e095c-ad94-495d-b6d3-aa6ea806257a-service-ca\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.481131 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.480920 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/427e095c-ad94-495d-b6d3-aa6ea806257a-oauth-serving-cert\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.481131 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.480946 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/427e095c-ad94-495d-b6d3-aa6ea806257a-console-serving-cert\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.481131 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.480980 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/427e095c-ad94-495d-b6d3-aa6ea806257a-console-config\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.481680 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.481648 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/427e095c-ad94-495d-b6d3-aa6ea806257a-oauth-serving-cert\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.481809 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.481684 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/427e095c-ad94-495d-b6d3-aa6ea806257a-service-ca\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.481809 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.481787 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/427e095c-ad94-495d-b6d3-aa6ea806257a-console-config\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.481809 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.481802 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/427e095c-ad94-495d-b6d3-aa6ea806257a-trusted-ca-bundle\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.483546 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.483521 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/427e095c-ad94-495d-b6d3-aa6ea806257a-console-oauth-config\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.483689 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.483669 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/427e095c-ad94-495d-b6d3-aa6ea806257a-console-serving-cert\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.489375 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.489353 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkhc6\" (UniqueName: \"kubernetes.io/projected/427e095c-ad94-495d-b6d3-aa6ea806257a-kube-api-access-tkhc6\") pod \"console-f6d755568-8gvcf\" (UID: \"427e095c-ad94-495d-b6d3-aa6ea806257a\") " pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.533418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.533388 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:11.670594 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:11.670560 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f6d755568-8gvcf"] Jul 10 17:02:11.672078 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:02:11.672048 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427e095c_ad94_495d_b6d3_aa6ea806257a.slice/crio-52f7089b242d68958af566a858daf0d3f16030b0a81ea72b71c85d472fc36ad2 WatchSource:0}: Error finding container 52f7089b242d68958af566a858daf0d3f16030b0a81ea72b71c85d472fc36ad2: Status 404 returned error can't find the container with id 52f7089b242d68958af566a858daf0d3f16030b0a81ea72b71c85d472fc36ad2 Jul 10 17:02:12.551063 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:12.551019 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f6d755568-8gvcf" event={"ID":"427e095c-ad94-495d-b6d3-aa6ea806257a","Type":"ContainerStarted","Data":"19cd1837336cedb9b461820be21361deaf4c7a7c7e45024e4f084a0a44f1f98e"} Jul 10 17:02:12.551561 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:12.551075 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f6d755568-8gvcf" event={"ID":"427e095c-ad94-495d-b6d3-aa6ea806257a","Type":"ContainerStarted","Data":"52f7089b242d68958af566a858daf0d3f16030b0a81ea72b71c85d472fc36ad2"} Jul 10 17:02:12.573704 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:12.573647 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f6d755568-8gvcf" podStartSLOduration=1.573628848 podStartE2EDuration="1.573628848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:02:12.5725421 +0000 UTC m=+462.459833423" watchObservedRunningTime="2026-07-10 17:02:12.573628848 +0000 UTC m=+462.460920168" Jul 10 17:02:15.071164 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:15.071131 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78"] Jul 10 17:02:15.073603 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:15.073586 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:15.077792 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:15.077760 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vcxl9\"" Jul 10 17:02:15.089930 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:15.089904 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78"] Jul 10 17:02:15.214999 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:15.214966 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/093d653c-8f1c-4841-a761-894cddb8b3c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9tp78\" (UID: \"093d653c-8f1c-4841-a761-894cddb8b3c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:15.215179 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:15.215016 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wdbm\" (UniqueName: \"kubernetes.io/projected/093d653c-8f1c-4841-a761-894cddb8b3c8-kube-api-access-5wdbm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9tp78\" (UID: \"093d653c-8f1c-4841-a761-894cddb8b3c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:15.315734 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:15.315702 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/093d653c-8f1c-4841-a761-894cddb8b3c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9tp78\" (UID: \"093d653c-8f1c-4841-a761-894cddb8b3c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:15.315734 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:15.315738 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wdbm\" (UniqueName: \"kubernetes.io/projected/093d653c-8f1c-4841-a761-894cddb8b3c8-kube-api-access-5wdbm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9tp78\" (UID: \"093d653c-8f1c-4841-a761-894cddb8b3c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:15.316107 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:15.316087 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/093d653c-8f1c-4841-a761-894cddb8b3c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9tp78\" (UID: \"093d653c-8f1c-4841-a761-894cddb8b3c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:15.324112 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:15.324043 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wdbm\" (UniqueName: \"kubernetes.io/projected/093d653c-8f1c-4841-a761-894cddb8b3c8-kube-api-access-5wdbm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9tp78\" (UID: \"093d653c-8f1c-4841-a761-894cddb8b3c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:15.383955 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:15.383921 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:15.517564 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:15.517536 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78"] Jul 10 17:02:15.520101 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:02:15.520077 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093d653c_8f1c_4841_a761_894cddb8b3c8.slice/crio-ef8b8d75076c27ef6f651b85342b6318b4a0b5a7296b725edd802cda15efaac6 WatchSource:0}: Error finding container ef8b8d75076c27ef6f651b85342b6318b4a0b5a7296b725edd802cda15efaac6: Status 404 returned error can't find the container with id ef8b8d75076c27ef6f651b85342b6318b4a0b5a7296b725edd802cda15efaac6 Jul 10 17:02:15.564019 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:15.563987 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" event={"ID":"093d653c-8f1c-4841-a761-894cddb8b3c8","Type":"ContainerStarted","Data":"ef8b8d75076c27ef6f651b85342b6318b4a0b5a7296b725edd802cda15efaac6"} Jul 10 17:02:17.015672 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:17.015629 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z"] Jul 10 17:02:17.020315 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:17.020266 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:17.022948 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:17.022913 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-h42sg\"" Jul 10 17:02:17.026220 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:17.026188 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2tz6\" (UniqueName: \"kubernetes.io/projected/d7f920f8-d068-4424-a5f5-b07160a324f6-kube-api-access-t2tz6\") pod \"limitador-operator-controller-manager-859d7b78bd-fbq8z\" (UID: \"d7f920f8-d068-4424-a5f5-b07160a324f6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:17.036554 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:17.036523 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z"] Jul 10 17:02:17.126768 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:17.126718 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2tz6\" (UniqueName: \"kubernetes.io/projected/d7f920f8-d068-4424-a5f5-b07160a324f6-kube-api-access-t2tz6\") pod \"limitador-operator-controller-manager-859d7b78bd-fbq8z\" (UID: \"d7f920f8-d068-4424-a5f5-b07160a324f6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:17.136402 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:17.136377 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2tz6\" (UniqueName: \"kubernetes.io/projected/d7f920f8-d068-4424-a5f5-b07160a324f6-kube-api-access-t2tz6\") pod \"limitador-operator-controller-manager-859d7b78bd-fbq8z\" (UID: \"d7f920f8-d068-4424-a5f5-b07160a324f6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:17.332618 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:17.332524 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:17.480004 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:17.479956 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z"] Jul 10 17:02:17.482763 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:02:17.482728 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f920f8_d068_4424_a5f5_b07160a324f6.slice/crio-c6308f780b6bb0073d04c9970fbcf973ec2292d9e3469a5ddc546e0f7ec334e1 WatchSource:0}: Error finding container c6308f780b6bb0073d04c9970fbcf973ec2292d9e3469a5ddc546e0f7ec334e1: Status 404 returned error can't find the container with id c6308f780b6bb0073d04c9970fbcf973ec2292d9e3469a5ddc546e0f7ec334e1 Jul 10 17:02:17.573582 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:17.573548 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" event={"ID":"d7f920f8-d068-4424-a5f5-b07160a324f6","Type":"ContainerStarted","Data":"c6308f780b6bb0073d04c9970fbcf973ec2292d9e3469a5ddc546e0f7ec334e1"} Jul 10 17:02:19.544545 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:19.544452 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nx4cp"] Jul 10 17:02:19.547887 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:19.547859 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nx4cp" Jul 10 17:02:19.550527 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:19.550502 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2jqn5\"" Jul 10 17:02:19.559613 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:19.559587 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nx4cp"] Jul 10 17:02:19.585962 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:19.585906 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" event={"ID":"d7f920f8-d068-4424-a5f5-b07160a324f6","Type":"ContainerStarted","Data":"2bf40f1c160b0a6fc377203b8952b30372b0ef6a22f0f05cf97181cc3efa16d6"} Jul 10 17:02:19.586148 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:19.586062 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:19.645335 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:19.645288 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j75b6\" (UniqueName: \"kubernetes.io/projected/670c3697-d489-43a4-a30e-1e166b2cff98-kube-api-access-j75b6\") pod \"authorino-operator-75cc8d5cb7-nx4cp\" (UID: \"670c3697-d489-43a4-a30e-1e166b2cff98\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nx4cp" Jul 10 17:02:19.746314 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:19.746260 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75b6\" (UniqueName: \"kubernetes.io/projected/670c3697-d489-43a4-a30e-1e166b2cff98-kube-api-access-j75b6\") pod \"authorino-operator-75cc8d5cb7-nx4cp\" (UID: \"670c3697-d489-43a4-a30e-1e166b2cff98\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nx4cp" Jul 10 17:02:19.757999 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:19.757969 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75b6\" (UniqueName: \"kubernetes.io/projected/670c3697-d489-43a4-a30e-1e166b2cff98-kube-api-access-j75b6\") pod \"authorino-operator-75cc8d5cb7-nx4cp\" (UID: \"670c3697-d489-43a4-a30e-1e166b2cff98\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nx4cp" Jul 10 17:02:19.866147 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:19.866055 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nx4cp" Jul 10 17:02:20.017744 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:20.017688 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" podStartSLOduration=2.22769078 podStartE2EDuration="4.017663878s" totalImagesPullingTime="1.789973098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:02:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:02:19.602976293 +0000 UTC m=+469.490267643" watchObservedRunningTime="2026-07-10 17:02:20.017663878 +0000 UTC m=+469.904955199" Jul 10 17:02:20.018649 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:20.018623 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nx4cp"] Jul 10 17:02:20.020799 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:02:20.020767 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670c3697_d489_43a4_a30e_1e166b2cff98.slice/crio-9d1be42b3724b0a4f9173fe2535c535f08989381684ab2e7d1e2a2f8de710996 WatchSource:0}: Error finding container 9d1be42b3724b0a4f9173fe2535c535f08989381684ab2e7d1e2a2f8de710996: Status 404 returned error can't find the container with id 9d1be42b3724b0a4f9173fe2535c535f08989381684ab2e7d1e2a2f8de710996 Jul 10 17:02:20.592743 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:20.592708 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nx4cp" event={"ID":"670c3697-d489-43a4-a30e-1e166b2cff98","Type":"ContainerStarted","Data":"9d1be42b3724b0a4f9173fe2535c535f08989381684ab2e7d1e2a2f8de710996"} Jul 10 17:02:21.533951 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:21.533912 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:21.534139 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:21.533990 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:21.539605 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:21.539581 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:21.603164 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:21.603130 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f6d755568-8gvcf" Jul 10 17:02:21.678285 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:21.678231 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66c876d8c5-xcqlw"] Jul 10 17:02:22.934727 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:22.934699 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq7z9"] Jul 10 17:02:22.939387 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:22.939364 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq7z9" Jul 10 17:02:22.941707 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:22.941685 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 17:02:22.941832 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:22.941812 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-d6k89\"" Jul 10 17:02:22.948376 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:22.948355 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq7z9"] Jul 10 17:02:22.973714 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:22.973687 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl27l\" (UniqueName: \"kubernetes.io/projected/7b8c8bfd-4bd4-4988-a44c-602b425174f0-kube-api-access-cl27l\") pod \"dns-operator-controller-manager-864dd78c4c-fq7z9\" (UID: \"7b8c8bfd-4bd4-4988-a44c-602b425174f0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq7z9" Jul 10 17:02:23.075318 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:23.075291 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl27l\" (UniqueName: \"kubernetes.io/projected/7b8c8bfd-4bd4-4988-a44c-602b425174f0-kube-api-access-cl27l\") pod \"dns-operator-controller-manager-864dd78c4c-fq7z9\" (UID: \"7b8c8bfd-4bd4-4988-a44c-602b425174f0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq7z9" Jul 10 17:02:23.084367 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:23.084340 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl27l\" (UniqueName: \"kubernetes.io/projected/7b8c8bfd-4bd4-4988-a44c-602b425174f0-kube-api-access-cl27l\") pod \"dns-operator-controller-manager-864dd78c4c-fq7z9\" (UID: \"7b8c8bfd-4bd4-4988-a44c-602b425174f0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq7z9" Jul 10 17:02:23.342952 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:23.342914 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq7z9" Jul 10 17:02:23.503217 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:23.503186 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq7z9"] Jul 10 17:02:23.506199 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:02:23.506160 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8c8bfd_4bd4_4988_a44c_602b425174f0.slice/crio-4ea04f0f6f181ba4db523f21d98f20bd64d27aea23d854dc284d3a2105c3f6ed WatchSource:0}: Error finding container 4ea04f0f6f181ba4db523f21d98f20bd64d27aea23d854dc284d3a2105c3f6ed: Status 404 returned error can't find the container with id 4ea04f0f6f181ba4db523f21d98f20bd64d27aea23d854dc284d3a2105c3f6ed Jul 10 17:02:23.607888 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:23.607803 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq7z9" event={"ID":"7b8c8bfd-4bd4-4988-a44c-602b425174f0","Type":"ContainerStarted","Data":"4ea04f0f6f181ba4db523f21d98f20bd64d27aea23d854dc284d3a2105c3f6ed"} Jul 10 17:02:23.609459 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:23.609434 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" event={"ID":"093d653c-8f1c-4841-a761-894cddb8b3c8","Type":"ContainerStarted","Data":"f38332d01a53e4487598fb762d8208c09181ff6e7b4c7ac719ea3712cc1a26bb"} Jul 10 17:02:23.609610 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:23.609511 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:23.616392 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:23.616366 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nx4cp" event={"ID":"670c3697-d489-43a4-a30e-1e166b2cff98","Type":"ContainerStarted","Data":"651cb279dc4a73f256633d2059f1a76f72dbff3e1b5b868a279ad978277f1998"} Jul 10 17:02:23.616522 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:23.616506 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nx4cp" Jul 10 17:02:23.635842 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:23.635797 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" podStartSLOduration=1.2734233019999999 podStartE2EDuration="8.635784681s" totalImagesPullingTime="7.362361379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:02:23.634726985 +0000 UTC m=+473.522018308" watchObservedRunningTime="2026-07-10 17:02:23.635784681 +0000 UTC m=+473.523076001" Jul 10 17:02:25.906581 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:25.906532 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nx4cp" podStartSLOduration=4.009659302 podStartE2EDuration="6.906511776s" totalImagesPullingTime="2.896852474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:02:23.649960596 +0000 UTC m=+473.537251940" watchObservedRunningTime="2026-07-10 17:02:25.906511776 +0000 UTC m=+475.793803098" Jul 10 17:02:25.907807 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:25.907787 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf"] Jul 10 17:02:25.912231 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:25.912215 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" Jul 10 17:02:25.915033 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:25.915014 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 17:02:25.915377 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:25.915104 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 17:02:25.915377 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:25.915163 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9g9z6\"" Jul 10 17:02:25.920005 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:25.919984 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf"] Jul 10 17:02:26.001857 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.001826 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d4964467-0c88-43f5-8fb3-ac8fb7fba9bc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pplmf\" (UID: \"d4964467-0c88-43f5-8fb3-ac8fb7fba9bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" Jul 10 17:02:26.002029 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.001863 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4964467-0c88-43f5-8fb3-ac8fb7fba9bc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pplmf\" (UID: \"d4964467-0c88-43f5-8fb3-ac8fb7fba9bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" Jul 10 17:02:26.002029 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.001881 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x82qn\" (UniqueName: \"kubernetes.io/projected/d4964467-0c88-43f5-8fb3-ac8fb7fba9bc-kube-api-access-x82qn\") pod \"kuadrant-console-plugin-66b5774985-pplmf\" (UID: \"d4964467-0c88-43f5-8fb3-ac8fb7fba9bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" Jul 10 17:02:26.103200 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.103107 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d4964467-0c88-43f5-8fb3-ac8fb7fba9bc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pplmf\" (UID: \"d4964467-0c88-43f5-8fb3-ac8fb7fba9bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" Jul 10 17:02:26.103200 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.103151 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4964467-0c88-43f5-8fb3-ac8fb7fba9bc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pplmf\" (UID: \"d4964467-0c88-43f5-8fb3-ac8fb7fba9bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" Jul 10 17:02:26.103200 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.103180 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x82qn\" (UniqueName: \"kubernetes.io/projected/d4964467-0c88-43f5-8fb3-ac8fb7fba9bc-kube-api-access-x82qn\") pod \"kuadrant-console-plugin-66b5774985-pplmf\" (UID: \"d4964467-0c88-43f5-8fb3-ac8fb7fba9bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" Jul 10 17:02:26.103505 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:26.103299 2567 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 10 17:02:26.103505 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:26.103382 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4964467-0c88-43f5-8fb3-ac8fb7fba9bc-plugin-serving-cert podName:d4964467-0c88-43f5-8fb3-ac8fb7fba9bc nodeName:}" failed. No retries permitted until 2026-07-10 17:02:26.60336643 +0000 UTC m=+476.490657730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/d4964467-0c88-43f5-8fb3-ac8fb7fba9bc-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-pplmf" (UID: "d4964467-0c88-43f5-8fb3-ac8fb7fba9bc") : secret "plugin-serving-cert" not found Jul 10 17:02:26.103792 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.103772 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d4964467-0c88-43f5-8fb3-ac8fb7fba9bc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pplmf\" (UID: \"d4964467-0c88-43f5-8fb3-ac8fb7fba9bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" Jul 10 17:02:26.113066 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.113043 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x82qn\" (UniqueName: \"kubernetes.io/projected/d4964467-0c88-43f5-8fb3-ac8fb7fba9bc-kube-api-access-x82qn\") pod \"kuadrant-console-plugin-66b5774985-pplmf\" (UID: \"d4964467-0c88-43f5-8fb3-ac8fb7fba9bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" Jul 10 17:02:26.608006 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.607968 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4964467-0c88-43f5-8fb3-ac8fb7fba9bc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pplmf\" (UID: \"d4964467-0c88-43f5-8fb3-ac8fb7fba9bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" Jul 10 17:02:26.610466 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.610444 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4964467-0c88-43f5-8fb3-ac8fb7fba9bc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pplmf\" (UID: \"d4964467-0c88-43f5-8fb3-ac8fb7fba9bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" Jul 10 17:02:26.634875 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.634847 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq7z9" event={"ID":"7b8c8bfd-4bd4-4988-a44c-602b425174f0","Type":"ContainerStarted","Data":"576aebcabe9ab94886aef60be3f0c813a648ed0554b6bd4a2464463ab9484aa3"} Jul 10 17:02:26.635049 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.635034 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq7z9" Jul 10 17:02:26.650063 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.650012 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq7z9" podStartSLOduration=2.315558169 podStartE2EDuration="4.649995517s" totalImagesPullingTime="2.334437348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:02:26.649921523 +0000 UTC m=+476.537212845" watchObservedRunningTime="2026-07-10 17:02:26.649995517 +0000 UTC m=+476.537286838" Jul 10 17:02:26.823329 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.823261 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" Jul 10 17:02:26.963043 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:26.963016 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf"] Jul 10 17:02:26.964932 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:02:26.964901 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4964467_0c88_43f5_8fb3_ac8fb7fba9bc.slice/crio-4c31035c5c33932041945acd92e03edda2ea8720efaf925b93f932d3367f9e5c WatchSource:0}: Error finding container 4c31035c5c33932041945acd92e03edda2ea8720efaf925b93f932d3367f9e5c: Status 404 returned error can't find the container with id 4c31035c5c33932041945acd92e03edda2ea8720efaf925b93f932d3367f9e5c Jul 10 17:02:27.645514 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:27.645477 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" event={"ID":"d4964467-0c88-43f5-8fb3-ac8fb7fba9bc","Type":"ContainerStarted","Data":"4c31035c5c33932041945acd92e03edda2ea8720efaf925b93f932d3367f9e5c"} Jul 10 17:02:30.594827 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:30.594798 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:34.624838 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:34.624805 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nx4cp" Jul 10 17:02:34.625219 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:34.624894 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:36.345154 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.345118 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp"] Jul 10 17:02:36.352018 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.351993 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:36.366152 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.366123 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp"] Jul 10 17:02:36.399103 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.394312 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89bfde31-6639-43e2-afed-f98e6e7b41da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-svwhp\" (UID: \"89bfde31-6639-43e2-afed-f98e6e7b41da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:36.399103 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.399079 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxdcf\" (UniqueName: \"kubernetes.io/projected/89bfde31-6639-43e2-afed-f98e6e7b41da-kube-api-access-pxdcf\") pod \"kuadrant-operator-controller-manager-55467d9f7-svwhp\" (UID: \"89bfde31-6639-43e2-afed-f98e6e7b41da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:36.464353 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.464316 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp"] Jul 10 17:02:36.464670 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:36.464640 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-pxdcf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" podUID="89bfde31-6639-43e2-afed-f98e6e7b41da" Jul 10 17:02:36.500536 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.500499 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89bfde31-6639-43e2-afed-f98e6e7b41da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-svwhp\" (UID: \"89bfde31-6639-43e2-afed-f98e6e7b41da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:36.500728 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.500567 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxdcf\" (UniqueName: \"kubernetes.io/projected/89bfde31-6639-43e2-afed-f98e6e7b41da-kube-api-access-pxdcf\") pod \"kuadrant-operator-controller-manager-55467d9f7-svwhp\" (UID: \"89bfde31-6639-43e2-afed-f98e6e7b41da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:36.501013 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.500927 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4"] Jul 10 17:02:36.501212 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.501180 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89bfde31-6639-43e2-afed-f98e6e7b41da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-svwhp\" (UID: \"89bfde31-6639-43e2-afed-f98e6e7b41da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:36.505190 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.505164 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:36.514255 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.514228 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxdcf\" (UniqueName: \"kubernetes.io/projected/89bfde31-6639-43e2-afed-f98e6e7b41da-kube-api-access-pxdcf\") pod \"kuadrant-operator-controller-manager-55467d9f7-svwhp\" (UID: \"89bfde31-6639-43e2-afed-f98e6e7b41da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:36.515109 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.515082 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4"] Jul 10 17:02:36.533545 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.533514 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp"] Jul 10 17:02:36.547306 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.547260 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp"] Jul 10 17:02:36.566426 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.566396 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78"] Jul 10 17:02:36.566683 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.566627 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" podUID="093d653c-8f1c-4841-a761-894cddb8b3c8" containerName="manager" containerID="cri-o://f38332d01a53e4487598fb762d8208c09181ff6e7b4c7ac719ea3712cc1a26bb" gracePeriod=2 Jul 10 17:02:36.572353 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.572327 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78"] Jul 10 17:02:36.588726 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.588696 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4"] Jul 10 17:02:36.589007 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:36.588976 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-r766x], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" podUID="343e047d-b34b-4b83-8cee-4e32ad559c8c" Jul 10 17:02:36.594084 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.594046 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc"] Jul 10 17:02:36.594572 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.594553 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093d653c-8f1c-4841-a761-894cddb8b3c8" containerName="manager" Jul 10 17:02:36.594572 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.594574 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="093d653c-8f1c-4841-a761-894cddb8b3c8" containerName="manager" Jul 10 17:02:36.594730 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.594637 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="093d653c-8f1c-4841-a761-894cddb8b3c8" containerName="manager" Jul 10 17:02:36.598725 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.598669 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4"] Jul 10 17:02:36.598829 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.598774 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" Jul 10 17:02:36.601032 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:36.600985 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9tp78\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="093d653c-8f1c-4841-a761-894cddb8b3c8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:36.601147 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.600992 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r766x\" (UniqueName: \"kubernetes.io/projected/343e047d-b34b-4b83-8cee-4e32ad559c8c-kube-api-access-r766x\") pod \"kuadrant-operator-controller-manager-86878c769f-mqph4\" (UID: \"343e047d-b34b-4b83-8cee-4e32ad559c8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:36.601213 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.601178 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/343e047d-b34b-4b83-8cee-4e32ad559c8c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mqph4\" (UID: \"343e047d-b34b-4b83-8cee-4e32ad559c8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:36.608539 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.608512 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc"] Jul 10 17:02:36.613297 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.613173 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z"] Jul 10 17:02:36.613413 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.613390 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" podUID="d7f920f8-d068-4424-a5f5-b07160a324f6" containerName="manager" containerID="cri-o://2bf40f1c160b0a6fc377203b8952b30372b0ef6a22f0f05cf97181cc3efa16d6" gracePeriod=2 Jul 10 17:02:36.627729 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:36.627689 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9tp78\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="093d653c-8f1c-4841-a761-894cddb8b3c8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:36.630647 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.629667 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv"] Jul 10 17:02:36.632059 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:36.632024 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fbq8z\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="d7f920f8-d068-4424-a5f5-b07160a324f6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:36.634216 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.634192 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z"] Jul 10 17:02:36.634350 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.634333 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" Jul 10 17:02:36.636440 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:36.636410 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fbq8z\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="d7f920f8-d068-4424-a5f5-b07160a324f6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:36.648303 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.648241 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv"] Jul 10 17:02:36.668753 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:36.668712 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9tp78\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="093d653c-8f1c-4841-a761-894cddb8b3c8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:36.684158 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.684126 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="343e047d-b34b-4b83-8cee-4e32ad559c8c" path="/var/lib/kubelet/pods/343e047d-b34b-4b83-8cee-4e32ad559c8c/volumes" Jul 10 17:02:36.696246 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.696220 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:36.696604 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.696240 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:36.702318 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.702243 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r766x\" (UniqueName: \"kubernetes.io/projected/343e047d-b34b-4b83-8cee-4e32ad559c8c-kube-api-access-r766x\") pod \"kuadrant-operator-controller-manager-86878c769f-mqph4\" (UID: \"343e047d-b34b-4b83-8cee-4e32ad559c8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:36.702418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.702335 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/343e047d-b34b-4b83-8cee-4e32ad559c8c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mqph4\" (UID: \"343e047d-b34b-4b83-8cee-4e32ad559c8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:36.702418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.702378 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbsbs\" (UniqueName: \"kubernetes.io/projected/c211ce14-67fd-4e23-853b-0c3004469c23-kube-api-access-hbsbs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9hcrc\" (UID: \"c211ce14-67fd-4e23-853b-0c3004469c23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" Jul 10 17:02:36.702530 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.702416 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ngdc\" (UniqueName: \"kubernetes.io/projected/a288f196-9b79-44fa-a961-97c54a6c179d-kube-api-access-8ngdc\") pod \"kuadrant-operator-controller-manager-86878c769f-rq7zv\" (UID: \"a288f196-9b79-44fa-a961-97c54a6c179d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" Jul 10 17:02:36.702530 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.702464 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a288f196-9b79-44fa-a961-97c54a6c179d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rq7zv\" (UID: \"a288f196-9b79-44fa-a961-97c54a6c179d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" Jul 10 17:02:36.702530 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.702513 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c211ce14-67fd-4e23-853b-0c3004469c23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9hcrc\" (UID: \"c211ce14-67fd-4e23-853b-0c3004469c23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" Jul 10 17:02:36.702824 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.702791 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/343e047d-b34b-4b83-8cee-4e32ad559c8c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mqph4\" (UID: \"343e047d-b34b-4b83-8cee-4e32ad559c8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:36.706735 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:36.706713 2567 projected.go:196] Error preparing data for projected volume kube-api-access-r766x for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-mqph4" not found Jul 10 17:02:36.706827 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:36.706786 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/343e047d-b34b-4b83-8cee-4e32ad559c8c-kube-api-access-r766x podName:343e047d-b34b-4b83-8cee-4e32ad559c8c nodeName:}" failed. No retries permitted until 2026-07-10 17:02:37.20676491 +0000 UTC m=+487.094056208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r766x" (UniqueName: "kubernetes.io/projected/343e047d-b34b-4b83-8cee-4e32ad559c8c-kube-api-access-r766x") pod "kuadrant-operator-controller-manager-86878c769f-mqph4" (UID: "343e047d-b34b-4b83-8cee-4e32ad559c8c") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-mqph4" not found Jul 10 17:02:36.714550 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.714530 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:36.716879 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:36.716842 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-mqph4\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="343e047d-b34b-4b83-8cee-4e32ad559c8c" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:36.717557 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.717536 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:36.719408 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:36.719384 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-mqph4\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="343e047d-b34b-4b83-8cee-4e32ad559c8c" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:36.721071 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:36.721049 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-svwhp\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="89bfde31-6639-43e2-afed-f98e6e7b41da" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:36.803726 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.803692 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89bfde31-6639-43e2-afed-f98e6e7b41da-extensions-socket-volume\") pod \"89bfde31-6639-43e2-afed-f98e6e7b41da\" (UID: \"89bfde31-6639-43e2-afed-f98e6e7b41da\") " Jul 10 17:02:36.803896 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.803736 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/343e047d-b34b-4b83-8cee-4e32ad559c8c-extensions-socket-volume\") pod \"343e047d-b34b-4b83-8cee-4e32ad559c8c\" (UID: \"343e047d-b34b-4b83-8cee-4e32ad559c8c\") " Jul 10 17:02:36.803896 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.803777 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxdcf\" (UniqueName: \"kubernetes.io/projected/89bfde31-6639-43e2-afed-f98e6e7b41da-kube-api-access-pxdcf\") pod \"89bfde31-6639-43e2-afed-f98e6e7b41da\" (UID: \"89bfde31-6639-43e2-afed-f98e6e7b41da\") " Jul 10 17:02:36.804003 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.803968 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbsbs\" (UniqueName: \"kubernetes.io/projected/c211ce14-67fd-4e23-853b-0c3004469c23-kube-api-access-hbsbs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9hcrc\" (UID: \"c211ce14-67fd-4e23-853b-0c3004469c23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" Jul 10 17:02:36.804003 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.803976 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343e047d-b34b-4b83-8cee-4e32ad559c8c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "343e047d-b34b-4b83-8cee-4e32ad559c8c" (UID: "343e047d-b34b-4b83-8cee-4e32ad559c8c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:02:36.804088 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.804004 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ngdc\" (UniqueName: \"kubernetes.io/projected/a288f196-9b79-44fa-a961-97c54a6c179d-kube-api-access-8ngdc\") pod \"kuadrant-operator-controller-manager-86878c769f-rq7zv\" (UID: \"a288f196-9b79-44fa-a961-97c54a6c179d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" Jul 10 17:02:36.804088 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.804059 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a288f196-9b79-44fa-a961-97c54a6c179d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rq7zv\" (UID: \"a288f196-9b79-44fa-a961-97c54a6c179d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" Jul 10 17:02:36.804088 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.803976 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bfde31-6639-43e2-afed-f98e6e7b41da-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "89bfde31-6639-43e2-afed-f98e6e7b41da" (UID: "89bfde31-6639-43e2-afed-f98e6e7b41da"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:02:36.804236 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.804204 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c211ce14-67fd-4e23-853b-0c3004469c23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9hcrc\" (UID: \"c211ce14-67fd-4e23-853b-0c3004469c23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" Jul 10 17:02:36.804346 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.804330 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r766x\" (UniqueName: \"kubernetes.io/projected/343e047d-b34b-4b83-8cee-4e32ad559c8c-kube-api-access-r766x\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:36.804417 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.804352 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89bfde31-6639-43e2-afed-f98e6e7b41da-extensions-socket-volume\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:36.804417 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.804368 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/343e047d-b34b-4b83-8cee-4e32ad559c8c-extensions-socket-volume\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:36.804605 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.804580 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a288f196-9b79-44fa-a961-97c54a6c179d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rq7zv\" (UID: \"a288f196-9b79-44fa-a961-97c54a6c179d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" Jul 10 17:02:36.804772 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.804749 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c211ce14-67fd-4e23-853b-0c3004469c23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9hcrc\" (UID: \"c211ce14-67fd-4e23-853b-0c3004469c23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" Jul 10 17:02:36.806363 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.806337 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bfde31-6639-43e2-afed-f98e6e7b41da-kube-api-access-pxdcf" (OuterVolumeSpecName: "kube-api-access-pxdcf") pod "89bfde31-6639-43e2-afed-f98e6e7b41da" (UID: "89bfde31-6639-43e2-afed-f98e6e7b41da"). InnerVolumeSpecName "kube-api-access-pxdcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:02:36.812577 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.812539 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ngdc\" (UniqueName: \"kubernetes.io/projected/a288f196-9b79-44fa-a961-97c54a6c179d-kube-api-access-8ngdc\") pod \"kuadrant-operator-controller-manager-86878c769f-rq7zv\" (UID: \"a288f196-9b79-44fa-a961-97c54a6c179d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" Jul 10 17:02:36.812661 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.812640 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbsbs\" (UniqueName: \"kubernetes.io/projected/c211ce14-67fd-4e23-853b-0c3004469c23-kube-api-access-hbsbs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9hcrc\" (UID: \"c211ce14-67fd-4e23-853b-0c3004469c23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" Jul 10 17:02:36.905808 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:36.905739 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxdcf\" (UniqueName: \"kubernetes.io/projected/89bfde31-6639-43e2-afed-f98e6e7b41da-kube-api-access-pxdcf\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:37.000445 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:37.000408 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" Jul 10 17:02:37.009455 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:37.009429 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" Jul 10 17:02:37.647714 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:37.647682 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq7z9" Jul 10 17:02:37.649848 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:37.649812 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-mqph4\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="343e047d-b34b-4b83-8cee-4e32ad559c8c" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:37.652235 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:37.652206 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-svwhp\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="89bfde31-6639-43e2-afed-f98e6e7b41da" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:37.700747 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:37.700717 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:37.700924 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:37.700757 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:37.705537 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:37.705506 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-mqph4\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="343e047d-b34b-4b83-8cee-4e32ad559c8c" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:37.707675 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:37.707634 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-svwhp\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="89bfde31-6639-43e2-afed-f98e6e7b41da" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:37.709803 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:37.709772 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-mqph4\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="343e047d-b34b-4b83-8cee-4e32ad559c8c" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:37.714367 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:37.714342 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-svwhp\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="89bfde31-6639-43e2-afed-f98e6e7b41da" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:37.716975 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:37.716943 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-mqph4\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="343e047d-b34b-4b83-8cee-4e32ad559c8c" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqph4" Jul 10 17:02:37.719709 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:37.719676 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-svwhp\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="89bfde31-6639-43e2-afed-f98e6e7b41da" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-svwhp" Jul 10 17:02:38.682436 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:38.682406 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="343e047d-b34b-4b83-8cee-4e32ad559c8c" path="/var/lib/kubelet/pods/343e047d-b34b-4b83-8cee-4e32ad559c8c/volumes" Jul 10 17:02:38.682868 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:38.682706 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89bfde31-6639-43e2-afed-f98e6e7b41da" path="/var/lib/kubelet/pods/89bfde31-6639-43e2-afed-f98e6e7b41da/volumes" Jul 10 17:02:46.710714 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:46.710655 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66c876d8c5-xcqlw" podUID="c59a60cb-69e9-43d3-a4b8-1981412f2ee3" containerName="console" containerID="cri-o://f6f2907695df39bff0f9b649c9ca6b84cf2fd466a6751925475ce238ee9f9873" gracePeriod=15 Jul 10 17:02:50.342488 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.342111 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:50.347721 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:50.347649 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fbq8z\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="d7f920f8-d068-4424-a5f5-b07160a324f6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:50.393717 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.393688 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 17:02:50.396110 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.396080 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:50.417256 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:50.417136 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fbq8z\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="d7f920f8-d068-4424-a5f5-b07160a324f6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:50.419177 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:50.419135 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fbq8z\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="d7f920f8-d068-4424-a5f5-b07160a324f6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:50.421327 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:50.421293 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9tp78\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="093d653c-8f1c-4841-a761-894cddb8b3c8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:50.427315 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.427263 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2tz6\" (UniqueName: \"kubernetes.io/projected/d7f920f8-d068-4424-a5f5-b07160a324f6-kube-api-access-t2tz6\") pod \"d7f920f8-d068-4424-a5f5-b07160a324f6\" (UID: \"d7f920f8-d068-4424-a5f5-b07160a324f6\") " Jul 10 17:02:50.429432 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.429405 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f920f8-d068-4424-a5f5-b07160a324f6-kube-api-access-t2tz6" (OuterVolumeSpecName: "kube-api-access-t2tz6") pod "d7f920f8-d068-4424-a5f5-b07160a324f6" (UID: "d7f920f8-d068-4424-a5f5-b07160a324f6"). InnerVolumeSpecName "kube-api-access-t2tz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:02:50.528753 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.528709 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/093d653c-8f1c-4841-a761-894cddb8b3c8-extensions-socket-volume\") pod \"093d653c-8f1c-4841-a761-894cddb8b3c8\" (UID: \"093d653c-8f1c-4841-a761-894cddb8b3c8\") " Jul 10 17:02:50.528753 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.528755 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-service-ca\") pod \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " Jul 10 17:02:50.529067 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.528780 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-config\") pod \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " Jul 10 17:02:50.529067 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.528820 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wdbm\" (UniqueName: \"kubernetes.io/projected/093d653c-8f1c-4841-a761-894cddb8b3c8-kube-api-access-5wdbm\") pod \"093d653c-8f1c-4841-a761-894cddb8b3c8\" (UID: \"093d653c-8f1c-4841-a761-894cddb8b3c8\") " Jul 10 17:02:50.529067 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.528850 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-serving-cert\") pod \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " Jul 10 17:02:50.529067 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.528889 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-trusted-ca-bundle\") pod \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " Jul 10 17:02:50.529067 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.528924 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgwrl\" (UniqueName: \"kubernetes.io/projected/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-kube-api-access-hgwrl\") pod \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " Jul 10 17:02:50.529067 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.528950 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-oauth-config\") pod \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " Jul 10 17:02:50.529067 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.529007 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-oauth-serving-cert\") pod \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\" (UID: \"c59a60cb-69e9-43d3-a4b8-1981412f2ee3\") " Jul 10 17:02:50.529504 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.529143 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093d653c-8f1c-4841-a761-894cddb8b3c8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "093d653c-8f1c-4841-a761-894cddb8b3c8" (UID: "093d653c-8f1c-4841-a761-894cddb8b3c8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:02:50.529504 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.529247 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-service-ca" (OuterVolumeSpecName: "service-ca") pod "c59a60cb-69e9-43d3-a4b8-1981412f2ee3" (UID: "c59a60cb-69e9-43d3-a4b8-1981412f2ee3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:02:50.529504 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.529261 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-config" (OuterVolumeSpecName: "console-config") pod "c59a60cb-69e9-43d3-a4b8-1981412f2ee3" (UID: "c59a60cb-69e9-43d3-a4b8-1981412f2ee3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:02:50.529504 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.529346 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2tz6\" (UniqueName: \"kubernetes.io/projected/d7f920f8-d068-4424-a5f5-b07160a324f6-kube-api-access-t2tz6\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:50.529504 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.529367 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/093d653c-8f1c-4841-a761-894cddb8b3c8-extensions-socket-volume\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:50.529703 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.529553 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c59a60cb-69e9-43d3-a4b8-1981412f2ee3" (UID: "c59a60cb-69e9-43d3-a4b8-1981412f2ee3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:02:50.529703 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.529598 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c59a60cb-69e9-43d3-a4b8-1981412f2ee3" (UID: "c59a60cb-69e9-43d3-a4b8-1981412f2ee3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:02:50.531251 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.531218 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c59a60cb-69e9-43d3-a4b8-1981412f2ee3" (UID: "c59a60cb-69e9-43d3-a4b8-1981412f2ee3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:02:50.531251 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.531224 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-kube-api-access-hgwrl" (OuterVolumeSpecName: "kube-api-access-hgwrl") pod "c59a60cb-69e9-43d3-a4b8-1981412f2ee3" (UID: "c59a60cb-69e9-43d3-a4b8-1981412f2ee3"). InnerVolumeSpecName "kube-api-access-hgwrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:02:50.531251 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.531234 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093d653c-8f1c-4841-a761-894cddb8b3c8-kube-api-access-5wdbm" (OuterVolumeSpecName: "kube-api-access-5wdbm") pod "093d653c-8f1c-4841-a761-894cddb8b3c8" (UID: "093d653c-8f1c-4841-a761-894cddb8b3c8"). InnerVolumeSpecName "kube-api-access-5wdbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:02:50.531469 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.531455 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c59a60cb-69e9-43d3-a4b8-1981412f2ee3" (UID: "c59a60cb-69e9-43d3-a4b8-1981412f2ee3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:02:50.600543 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.600514 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv"] Jul 10 17:02:50.609485 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.605202 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc"] Jul 10 17:02:50.609485 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:02:50.605415 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda288f196_9b79_44fa_a961_97c54a6c179d.slice/crio-5e1c509801236de5e74d75d15b847d8b8993c206bf92b47302d62864f872224b WatchSource:0}: Error finding container 5e1c509801236de5e74d75d15b847d8b8993c206bf92b47302d62864f872224b: Status 404 returned error can't find the container with id 5e1c509801236de5e74d75d15b847d8b8993c206bf92b47302d62864f872224b Jul 10 17:02:50.630144 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.630121 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-trusted-ca-bundle\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:50.630144 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.630144 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgwrl\" (UniqueName: \"kubernetes.io/projected/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-kube-api-access-hgwrl\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:50.630314 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.630154 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-oauth-config\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:50.630314 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.630163 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-oauth-serving-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:50.630314 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.630171 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:50.630314 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.630179 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-config\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:50.630314 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.630187 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wdbm\" (UniqueName: \"kubernetes.io/projected/093d653c-8f1c-4841-a761-894cddb8b3c8-kube-api-access-5wdbm\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:50.630314 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.630195 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c59a60cb-69e9-43d3-a4b8-1981412f2ee3-console-serving-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:02:50.680686 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:50.680637 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9tp78\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="093d653c-8f1c-4841-a761-894cddb8b3c8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:50.682366 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:50.682330 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fbq8z\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="d7f920f8-d068-4424-a5f5-b07160a324f6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:50.683338 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.683319 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="093d653c-8f1c-4841-a761-894cddb8b3c8" path="/var/lib/kubelet/pods/093d653c-8f1c-4841-a761-894cddb8b3c8/volumes" Jul 10 17:02:50.683698 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.683680 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7f920f8-d068-4424-a5f5-b07160a324f6" path="/var/lib/kubelet/pods/d7f920f8-d068-4424-a5f5-b07160a324f6/volumes" Jul 10 17:02:50.767265 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.767239 2567 generic.go:356] "Generic (PLEG): container finished" podID="c59a60cb-69e9-43d3-a4b8-1981412f2ee3" containerID="f6f2907695df39bff0f9b649c9ca6b84cf2fd466a6751925475ce238ee9f9873" exitCode=0 Jul 10 17:02:50.767425 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.767314 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c876d8c5-xcqlw" Jul 10 17:02:50.767425 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.767310 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c876d8c5-xcqlw" event={"ID":"c59a60cb-69e9-43d3-a4b8-1981412f2ee3","Type":"ContainerDied","Data":"f6f2907695df39bff0f9b649c9ca6b84cf2fd466a6751925475ce238ee9f9873"} Jul 10 17:02:50.767515 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.767460 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c876d8c5-xcqlw" event={"ID":"c59a60cb-69e9-43d3-a4b8-1981412f2ee3","Type":"ContainerDied","Data":"a9a12ddd6efea544933eb8b1c2f724d35de96d03deed622213e016716d1a19b3"} Jul 10 17:02:50.767515 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.767497 2567 scope.go:117] "RemoveContainer" containerID="f6f2907695df39bff0f9b649c9ca6b84cf2fd466a6751925475ce238ee9f9873" Jul 10 17:02:50.768714 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.768695 2567 generic.go:356] "Generic (PLEG): container finished" podID="d7f920f8-d068-4424-a5f5-b07160a324f6" containerID="2bf40f1c160b0a6fc377203b8952b30372b0ef6a22f0f05cf97181cc3efa16d6" exitCode=0 Jul 10 17:02:50.768810 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.768761 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:50.770700 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.770653 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" event={"ID":"d4964467-0c88-43f5-8fb3-ac8fb7fba9bc","Type":"ContainerStarted","Data":"f9bb4cc82eee912da9ec60d842c3bb22c30983f9501481af0e2179dfaebe80d2"} Jul 10 17:02:50.772209 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.772190 2567 generic.go:356] "Generic (PLEG): container finished" podID="093d653c-8f1c-4841-a761-894cddb8b3c8" containerID="f38332d01a53e4487598fb762d8208c09181ff6e7b4c7ac719ea3712cc1a26bb" exitCode=0 Jul 10 17:02:50.772333 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.772239 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:50.774010 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:50.773978 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fbq8z\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="d7f920f8-d068-4424-a5f5-b07160a324f6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:50.774880 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.774851 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" event={"ID":"a288f196-9b79-44fa-a961-97c54a6c179d","Type":"ContainerStarted","Data":"044cfb9b0c83d64c2f7f2a33a387771470c819744a800550054c7cf087d1788b"} Jul 10 17:02:50.774979 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.774890 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" event={"ID":"a288f196-9b79-44fa-a961-97c54a6c179d","Type":"ContainerStarted","Data":"5e1c509801236de5e74d75d15b847d8b8993c206bf92b47302d62864f872224b"} Jul 10 17:02:50.774979 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.774934 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" Jul 10 17:02:50.776675 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.776652 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" event={"ID":"c211ce14-67fd-4e23-853b-0c3004469c23","Type":"ContainerStarted","Data":"565b1c4f80cc9a5d112817a16c28bd034c33e5c8f8ddcb5a8568e8dfec0db1aa"} Jul 10 17:02:50.776753 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.776695 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" event={"ID":"c211ce14-67fd-4e23-853b-0c3004469c23","Type":"ContainerStarted","Data":"a6b4f3806a87e01ea6ac80ce1dbe12981228451e28aa9f615277bd2183282909"} Jul 10 17:02:50.776820 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.776806 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" Jul 10 17:02:50.778418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.778396 2567 scope.go:117] "RemoveContainer" containerID="f6f2907695df39bff0f9b649c9ca6b84cf2fd466a6751925475ce238ee9f9873" Jul 10 17:02:50.778712 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:50.778687 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f2907695df39bff0f9b649c9ca6b84cf2fd466a6751925475ce238ee9f9873\": container with ID starting with f6f2907695df39bff0f9b649c9ca6b84cf2fd466a6751925475ce238ee9f9873 not found: ID does not exist" containerID="f6f2907695df39bff0f9b649c9ca6b84cf2fd466a6751925475ce238ee9f9873" Jul 10 17:02:50.778800 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.778721 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f2907695df39bff0f9b649c9ca6b84cf2fd466a6751925475ce238ee9f9873"} err="failed to get container status \"f6f2907695df39bff0f9b649c9ca6b84cf2fd466a6751925475ce238ee9f9873\": rpc error: code = NotFound desc = could not find container \"f6f2907695df39bff0f9b649c9ca6b84cf2fd466a6751925475ce238ee9f9873\": container with ID starting with f6f2907695df39bff0f9b649c9ca6b84cf2fd466a6751925475ce238ee9f9873 not found: ID does not exist" Jul 10 17:02:50.778800 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.778743 2567 scope.go:117] "RemoveContainer" containerID="2bf40f1c160b0a6fc377203b8952b30372b0ef6a22f0f05cf97181cc3efa16d6" Jul 10 17:02:50.787169 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.787153 2567 scope.go:117] "RemoveContainer" containerID="2bf40f1c160b0a6fc377203b8952b30372b0ef6a22f0f05cf97181cc3efa16d6" Jul 10 17:02:50.787498 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:50.787467 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf40f1c160b0a6fc377203b8952b30372b0ef6a22f0f05cf97181cc3efa16d6\": container with ID starting with 2bf40f1c160b0a6fc377203b8952b30372b0ef6a22f0f05cf97181cc3efa16d6 not found: ID does not exist" containerID="2bf40f1c160b0a6fc377203b8952b30372b0ef6a22f0f05cf97181cc3efa16d6" Jul 10 17:02:50.787589 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.787492 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf40f1c160b0a6fc377203b8952b30372b0ef6a22f0f05cf97181cc3efa16d6"} err="failed to get container status \"2bf40f1c160b0a6fc377203b8952b30372b0ef6a22f0f05cf97181cc3efa16d6\": rpc error: code = NotFound desc = could not find container \"2bf40f1c160b0a6fc377203b8952b30372b0ef6a22f0f05cf97181cc3efa16d6\": container with ID starting with 2bf40f1c160b0a6fc377203b8952b30372b0ef6a22f0f05cf97181cc3efa16d6 not found: ID does not exist" Jul 10 17:02:50.787589 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.787511 2567 scope.go:117] "RemoveContainer" containerID="f38332d01a53e4487598fb762d8208c09181ff6e7b4c7ac719ea3712cc1a26bb" Jul 10 17:02:50.794076 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.794027 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pplmf" podStartSLOduration=2.490918223 podStartE2EDuration="25.794010806s" totalImagesPullingTime="23.303092583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:02:50.792008844 +0000 UTC m=+500.679300186" watchObservedRunningTime="2026-07-10 17:02:50.794010806 +0000 UTC m=+500.681302128" Jul 10 17:02:50.794340 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:50.794311 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fbq8z\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="d7f920f8-d068-4424-a5f5-b07160a324f6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbq8z" Jul 10 17:02:50.796102 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.796086 2567 scope.go:117] "RemoveContainer" containerID="f38332d01a53e4487598fb762d8208c09181ff6e7b4c7ac719ea3712cc1a26bb" Jul 10 17:02:50.796362 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:50.796342 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f38332d01a53e4487598fb762d8208c09181ff6e7b4c7ac719ea3712cc1a26bb\": container with ID starting with f38332d01a53e4487598fb762d8208c09181ff6e7b4c7ac719ea3712cc1a26bb not found: ID does not exist" containerID="f38332d01a53e4487598fb762d8208c09181ff6e7b4c7ac719ea3712cc1a26bb" Jul 10 17:02:50.796419 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.796368 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38332d01a53e4487598fb762d8208c09181ff6e7b4c7ac719ea3712cc1a26bb"} err="failed to get container status \"f38332d01a53e4487598fb762d8208c09181ff6e7b4c7ac719ea3712cc1a26bb\": rpc error: code = NotFound desc = could not find container \"f38332d01a53e4487598fb762d8208c09181ff6e7b4c7ac719ea3712cc1a26bb\": container with ID starting with f38332d01a53e4487598fb762d8208c09181ff6e7b4c7ac719ea3712cc1a26bb not found: ID does not exist" Jul 10 17:02:50.818215 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.818128 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" podStartSLOduration=14.818111696999999 podStartE2EDuration="14.818111697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:02:50.817760103 +0000 UTC m=+500.705051442" watchObservedRunningTime="2026-07-10 17:02:50.818111697 +0000 UTC m=+500.705403018" Jul 10 17:02:50.832042 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.832015 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66c876d8c5-xcqlw"] Jul 10 17:02:50.834887 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.834865 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66c876d8c5-xcqlw"] Jul 10 17:02:50.850826 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:50.850785 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" podStartSLOduration=14.850771435 podStartE2EDuration="14.850771435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:02:50.849729666 +0000 UTC m=+500.737021010" watchObservedRunningTime="2026-07-10 17:02:50.850771435 +0000 UTC m=+500.738062754" Jul 10 17:02:50.851949 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:02:50.851927 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9tp78\" is forbidden: User \"system:node:ip-10-0-131-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-207.ec2.internal' and this object" podUID="093d653c-8f1c-4841-a761-894cddb8b3c8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9tp78" Jul 10 17:02:52.682166 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:02:52.682133 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c59a60cb-69e9-43d3-a4b8-1981412f2ee3" path="/var/lib/kubelet/pods/c59a60cb-69e9-43d3-a4b8-1981412f2ee3/volumes" Jul 10 17:03:01.787253 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:01.787220 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" Jul 10 17:03:01.787817 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:01.787321 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" Jul 10 17:03:01.865521 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:01.865488 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc"] Jul 10 17:03:01.865717 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:01.865690 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" podUID="c211ce14-67fd-4e23-853b-0c3004469c23" containerName="manager" containerID="cri-o://565b1c4f80cc9a5d112817a16c28bd034c33e5c8f8ddcb5a8568e8dfec0db1aa" gracePeriod=10 Jul 10 17:03:02.630894 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.630867 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" Jul 10 17:03:02.731589 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.731554 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbsbs\" (UniqueName: \"kubernetes.io/projected/c211ce14-67fd-4e23-853b-0c3004469c23-kube-api-access-hbsbs\") pod \"c211ce14-67fd-4e23-853b-0c3004469c23\" (UID: \"c211ce14-67fd-4e23-853b-0c3004469c23\") " Jul 10 17:03:02.731787 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.731624 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c211ce14-67fd-4e23-853b-0c3004469c23-extensions-socket-volume\") pod \"c211ce14-67fd-4e23-853b-0c3004469c23\" (UID: \"c211ce14-67fd-4e23-853b-0c3004469c23\") " Jul 10 17:03:02.731950 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.731920 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c211ce14-67fd-4e23-853b-0c3004469c23-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c211ce14-67fd-4e23-853b-0c3004469c23" (UID: "c211ce14-67fd-4e23-853b-0c3004469c23"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:03:02.733892 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.733869 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c211ce14-67fd-4e23-853b-0c3004469c23-kube-api-access-hbsbs" (OuterVolumeSpecName: "kube-api-access-hbsbs") pod "c211ce14-67fd-4e23-853b-0c3004469c23" (UID: "c211ce14-67fd-4e23-853b-0c3004469c23"). InnerVolumeSpecName "kube-api-access-hbsbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:03:02.833151 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.833072 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c211ce14-67fd-4e23-853b-0c3004469c23-extensions-socket-volume\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:03:02.833151 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.833104 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbsbs\" (UniqueName: \"kubernetes.io/projected/c211ce14-67fd-4e23-853b-0c3004469c23-kube-api-access-hbsbs\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:03:02.838329 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.838291 2567 generic.go:356] "Generic (PLEG): container finished" podID="c211ce14-67fd-4e23-853b-0c3004469c23" containerID="565b1c4f80cc9a5d112817a16c28bd034c33e5c8f8ddcb5a8568e8dfec0db1aa" exitCode=0 Jul 10 17:03:02.838503 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.838340 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" Jul 10 17:03:02.838503 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.838342 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" event={"ID":"c211ce14-67fd-4e23-853b-0c3004469c23","Type":"ContainerDied","Data":"565b1c4f80cc9a5d112817a16c28bd034c33e5c8f8ddcb5a8568e8dfec0db1aa"} Jul 10 17:03:02.838503 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.838479 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc" event={"ID":"c211ce14-67fd-4e23-853b-0c3004469c23","Type":"ContainerDied","Data":"a6b4f3806a87e01ea6ac80ce1dbe12981228451e28aa9f615277bd2183282909"} Jul 10 17:03:02.838666 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.838508 2567 scope.go:117] "RemoveContainer" containerID="565b1c4f80cc9a5d112817a16c28bd034c33e5c8f8ddcb5a8568e8dfec0db1aa" Jul 10 17:03:02.847943 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.847922 2567 scope.go:117] "RemoveContainer" containerID="565b1c4f80cc9a5d112817a16c28bd034c33e5c8f8ddcb5a8568e8dfec0db1aa" Jul 10 17:03:02.848210 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:03:02.848192 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565b1c4f80cc9a5d112817a16c28bd034c33e5c8f8ddcb5a8568e8dfec0db1aa\": container with ID starting with 565b1c4f80cc9a5d112817a16c28bd034c33e5c8f8ddcb5a8568e8dfec0db1aa not found: ID does not exist" containerID="565b1c4f80cc9a5d112817a16c28bd034c33e5c8f8ddcb5a8568e8dfec0db1aa" Jul 10 17:03:02.848260 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.848219 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565b1c4f80cc9a5d112817a16c28bd034c33e5c8f8ddcb5a8568e8dfec0db1aa"} err="failed to get container status \"565b1c4f80cc9a5d112817a16c28bd034c33e5c8f8ddcb5a8568e8dfec0db1aa\": rpc error: code = NotFound desc = could not find container \"565b1c4f80cc9a5d112817a16c28bd034c33e5c8f8ddcb5a8568e8dfec0db1aa\": container with ID starting with 565b1c4f80cc9a5d112817a16c28bd034c33e5c8f8ddcb5a8568e8dfec0db1aa not found: ID does not exist" Jul 10 17:03:02.859177 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.859154 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc"] Jul 10 17:03:02.865426 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:02.865404 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9hcrc"] Jul 10 17:03:04.681544 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:04.681511 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c211ce14-67fd-4e23-853b-0c3004469c23" path="/var/lib/kubelet/pods/c211ce14-67fd-4e23-853b-0c3004469c23/volumes" Jul 10 17:03:18.280544 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.280515 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph"] Jul 10 17:03:18.281015 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.280826 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7f920f8-d068-4424-a5f5-b07160a324f6" containerName="manager" Jul 10 17:03:18.281015 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.280836 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7f920f8-d068-4424-a5f5-b07160a324f6" containerName="manager" Jul 10 17:03:18.281015 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.280849 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c59a60cb-69e9-43d3-a4b8-1981412f2ee3" containerName="console" Jul 10 17:03:18.281015 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.280854 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c59a60cb-69e9-43d3-a4b8-1981412f2ee3" containerName="console" Jul 10 17:03:18.281015 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.280863 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c211ce14-67fd-4e23-853b-0c3004469c23" containerName="manager" Jul 10 17:03:18.281015 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.280868 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c211ce14-67fd-4e23-853b-0c3004469c23" containerName="manager" Jul 10 17:03:18.281015 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.280926 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7f920f8-d068-4424-a5f5-b07160a324f6" containerName="manager" Jul 10 17:03:18.281015 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.280932 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c211ce14-67fd-4e23-853b-0c3004469c23" containerName="manager" Jul 10 17:03:18.281015 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.280938 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c59a60cb-69e9-43d3-a4b8-1981412f2ee3" containerName="console" Jul 10 17:03:18.318542 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.318511 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph"] Jul 10 17:03:18.318689 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.318632 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.321151 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.321127 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 17:03:18.321313 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.321175 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ghq5v\"" Jul 10 17:03:18.352334 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.352311 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.352459 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.352343 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.352459 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.352360 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b2410-1f24-432b-9437-b7010fe00be2-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.352459 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.352412 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b2410-1f24-432b-9437-b7010fe00be2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.352578 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.352464 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8b6q\" (UniqueName: \"kubernetes.io/projected/f43b2410-1f24-432b-9437-b7010fe00be2-kube-api-access-p8b6q\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.352578 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.352493 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b2410-1f24-432b-9437-b7010fe00be2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.352578 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.352516 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.352578 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.352553 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.352578 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.352568 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.457045 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.457004 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8b6q\" (UniqueName: \"kubernetes.io/projected/f43b2410-1f24-432b-9437-b7010fe00be2-kube-api-access-p8b6q\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.457207 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.457064 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b2410-1f24-432b-9437-b7010fe00be2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.457207 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.457106 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.457207 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.457161 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.457207 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.457187 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.457438 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.457230 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.457438 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.457259 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.457438 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.457328 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b2410-1f24-432b-9437-b7010fe00be2-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.457438 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.457386 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b2410-1f24-432b-9437-b7010fe00be2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.457716 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.457691 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.457831 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.457721 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.457932 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.457768 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.458010 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.457884 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.458195 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.458147 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b2410-1f24-432b-9437-b7010fe00be2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.460106 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.460082 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b2410-1f24-432b-9437-b7010fe00be2-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.460201 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.460129 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b2410-1f24-432b-9437-b7010fe00be2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.466517 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.466485 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b2410-1f24-432b-9437-b7010fe00be2-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.466632 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.466536 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8b6q\" (UniqueName: \"kubernetes.io/projected/f43b2410-1f24-432b-9437-b7010fe00be2-kube-api-access-p8b6q\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7rph\" (UID: \"f43b2410-1f24-432b-9437-b7010fe00be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.628495 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.628399 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:18.775682 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.775654 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph"] Jul 10 17:03:18.778510 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:03:18.778476 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43b2410_1f24_432b_9437_b7010fe00be2.slice/crio-ec69e1c23752694c77858552e907be76783e7ae234754b69440f8a55e0028fbf WatchSource:0}: Error finding container ec69e1c23752694c77858552e907be76783e7ae234754b69440f8a55e0028fbf: Status 404 returned error can't find the container with id ec69e1c23752694c77858552e907be76783e7ae234754b69440f8a55e0028fbf Jul 10 17:03:18.910113 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:18.910031 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" event={"ID":"f43b2410-1f24-432b-9437-b7010fe00be2","Type":"ContainerStarted","Data":"ec69e1c23752694c77858552e907be76783e7ae234754b69440f8a55e0028fbf"} Jul 10 17:03:21.599091 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:21.599051 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:03:21.599414 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:21.599129 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:03:21.599414 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:21.599154 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:03:21.926607 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:21.926521 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" event={"ID":"f43b2410-1f24-432b-9437-b7010fe00be2","Type":"ContainerStarted","Data":"566efa65d07bb4786cbc66f0ed36d7d302ed034b6f216ad53a55de20f20e76d8"} Jul 10 17:03:21.952439 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:21.952382 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" podStartSLOduration=1.134141325 podStartE2EDuration="3.952367563s" totalImagesPullingTime="2.818226238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:03:21.950407491 +0000 UTC m=+531.837698811" watchObservedRunningTime="2026-07-10 17:03:21.952367563 +0000 UTC m=+531.839658927" Jul 10 17:03:22.628912 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:22.628884 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:22.633520 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:22.633495 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:22.931735 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:22.931659 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:22.932539 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:22.932518 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:03:23.091550 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.091515 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-89qd9"] Jul 10 17:03:23.093919 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.093902 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" Jul 10 17:03:23.096490 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.096466 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 17:03:23.105193 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.105169 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-89qd9"] Jul 10 17:03:23.189632 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.189539 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-89qd9"] Jul 10 17:03:23.200189 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.200149 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/03e6ef32-7e75-4e08-acd3-17540daea739-config-file\") pod \"limitador-limitador-64c94fd64f-89qd9\" (UID: \"03e6ef32-7e75-4e08-acd3-17540daea739\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" Jul 10 17:03:23.200398 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.200198 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z47g5\" (UniqueName: \"kubernetes.io/projected/03e6ef32-7e75-4e08-acd3-17540daea739-kube-api-access-z47g5\") pod \"limitador-limitador-64c94fd64f-89qd9\" (UID: \"03e6ef32-7e75-4e08-acd3-17540daea739\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" Jul 10 17:03:23.301112 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.301072 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/03e6ef32-7e75-4e08-acd3-17540daea739-config-file\") pod \"limitador-limitador-64c94fd64f-89qd9\" (UID: \"03e6ef32-7e75-4e08-acd3-17540daea739\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" Jul 10 17:03:23.301112 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.301113 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z47g5\" (UniqueName: \"kubernetes.io/projected/03e6ef32-7e75-4e08-acd3-17540daea739-kube-api-access-z47g5\") pod \"limitador-limitador-64c94fd64f-89qd9\" (UID: \"03e6ef32-7e75-4e08-acd3-17540daea739\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" Jul 10 17:03:23.301847 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.301825 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/03e6ef32-7e75-4e08-acd3-17540daea739-config-file\") pod \"limitador-limitador-64c94fd64f-89qd9\" (UID: \"03e6ef32-7e75-4e08-acd3-17540daea739\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" Jul 10 17:03:23.308646 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.308616 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z47g5\" (UniqueName: \"kubernetes.io/projected/03e6ef32-7e75-4e08-acd3-17540daea739-kube-api-access-z47g5\") pod \"limitador-limitador-64c94fd64f-89qd9\" (UID: \"03e6ef32-7e75-4e08-acd3-17540daea739\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" Jul 10 17:03:23.404281 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.404246 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" Jul 10 17:03:23.536515 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.536491 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-89qd9"] Jul 10 17:03:23.538699 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:03:23.538671 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e6ef32_7e75_4e08_acd3_17540daea739.slice/crio-e2cdb9dfe6c97a4fe48ff29bba2e7b1808f562e03312b6e64b8a0a51a766d8a6 WatchSource:0}: Error finding container e2cdb9dfe6c97a4fe48ff29bba2e7b1808f562e03312b6e64b8a0a51a766d8a6: Status 404 returned error can't find the container with id e2cdb9dfe6c97a4fe48ff29bba2e7b1808f562e03312b6e64b8a0a51a766d8a6 Jul 10 17:03:23.923311 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.923221 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-h466f"] Jul 10 17:03:23.926552 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.926530 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-h466f" Jul 10 17:03:23.928778 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.928755 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6qrq2\"" Jul 10 17:03:23.934431 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.933632 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-h466f"] Jul 10 17:03:23.938839 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:23.938811 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" event={"ID":"03e6ef32-7e75-4e08-acd3-17540daea739","Type":"ContainerStarted","Data":"e2cdb9dfe6c97a4fe48ff29bba2e7b1808f562e03312b6e64b8a0a51a766d8a6"} Jul 10 17:03:24.006467 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.006435 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hbv9\" (UniqueName: \"kubernetes.io/projected/5159719d-19c8-4316-b6d3-549f5b09ec52-kube-api-access-9hbv9\") pod \"authorino-58cd955675-h466f\" (UID: \"5159719d-19c8-4316-b6d3-549f5b09ec52\") " pod="kuadrant-system/authorino-58cd955675-h466f" Jul 10 17:03:24.046484 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.046449 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-dt9wv"] Jul 10 17:03:24.048900 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.048878 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dt9wv" Jul 10 17:03:24.055910 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.055890 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dt9wv"] Jul 10 17:03:24.107142 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.107110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwcc7\" (UniqueName: \"kubernetes.io/projected/322d75e8-4004-4f21-983e-fa05d2833ece-kube-api-access-kwcc7\") pod \"authorino-75d556bd77-dt9wv\" (UID: \"322d75e8-4004-4f21-983e-fa05d2833ece\") " pod="kuadrant-system/authorino-75d556bd77-dt9wv" Jul 10 17:03:24.107342 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.107165 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hbv9\" (UniqueName: \"kubernetes.io/projected/5159719d-19c8-4316-b6d3-549f5b09ec52-kube-api-access-9hbv9\") pod \"authorino-58cd955675-h466f\" (UID: \"5159719d-19c8-4316-b6d3-549f5b09ec52\") " pod="kuadrant-system/authorino-58cd955675-h466f" Jul 10 17:03:24.115117 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.115094 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hbv9\" (UniqueName: \"kubernetes.io/projected/5159719d-19c8-4316-b6d3-549f5b09ec52-kube-api-access-9hbv9\") pod \"authorino-58cd955675-h466f\" (UID: \"5159719d-19c8-4316-b6d3-549f5b09ec52\") " pod="kuadrant-system/authorino-58cd955675-h466f" Jul 10 17:03:24.207900 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.207865 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwcc7\" (UniqueName: \"kubernetes.io/projected/322d75e8-4004-4f21-983e-fa05d2833ece-kube-api-access-kwcc7\") pod \"authorino-75d556bd77-dt9wv\" (UID: \"322d75e8-4004-4f21-983e-fa05d2833ece\") " pod="kuadrant-system/authorino-75d556bd77-dt9wv" Jul 10 17:03:24.216651 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.216627 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwcc7\" (UniqueName: \"kubernetes.io/projected/322d75e8-4004-4f21-983e-fa05d2833ece-kube-api-access-kwcc7\") pod \"authorino-75d556bd77-dt9wv\" (UID: \"322d75e8-4004-4f21-983e-fa05d2833ece\") " pod="kuadrant-system/authorino-75d556bd77-dt9wv" Jul 10 17:03:24.237404 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.237371 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-h466f" Jul 10 17:03:24.358585 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.358533 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dt9wv" Jul 10 17:03:24.426232 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.426089 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-h466f"] Jul 10 17:03:24.535245 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.535190 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dt9wv"] Jul 10 17:03:24.538336 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:03:24.538306 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod322d75e8_4004_4f21_983e_fa05d2833ece.slice/crio-e6b050a84eae6130097570cd3c1b9ad9d7910b8b679fbc1ca586245d376a662a WatchSource:0}: Error finding container e6b050a84eae6130097570cd3c1b9ad9d7910b8b679fbc1ca586245d376a662a: Status 404 returned error can't find the container with id e6b050a84eae6130097570cd3c1b9ad9d7910b8b679fbc1ca586245d376a662a Jul 10 17:03:24.946213 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.946173 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dt9wv" event={"ID":"322d75e8-4004-4f21-983e-fa05d2833ece","Type":"ContainerStarted","Data":"e6b050a84eae6130097570cd3c1b9ad9d7910b8b679fbc1ca586245d376a662a"} Jul 10 17:03:24.947577 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:24.947551 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-h466f" event={"ID":"5159719d-19c8-4316-b6d3-549f5b09ec52","Type":"ContainerStarted","Data":"335176671c3d78b7e5e974271edddd12f1cfa3655701b6cd4fb1628bb5846954"} Jul 10 17:03:28.976618 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:28.976569 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-h466f" event={"ID":"5159719d-19c8-4316-b6d3-549f5b09ec52","Type":"ContainerStarted","Data":"12b22d04befbdf166bc022f205bf6fc66698a2d6a3e6bdf1a527d1a59c0613d7"} Jul 10 17:03:28.978087 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:28.978060 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" event={"ID":"03e6ef32-7e75-4e08-acd3-17540daea739","Type":"ContainerStarted","Data":"18bafdeab1f1a8254a3f597c5b99667172f1e72928b5efd8f7c1d5344d789695"} Jul 10 17:03:28.978217 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:28.978151 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" Jul 10 17:03:28.979500 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:28.979479 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dt9wv" event={"ID":"322d75e8-4004-4f21-983e-fa05d2833ece","Type":"ContainerStarted","Data":"089a6001091037b28ed6b102e07d7f04746ba01c74264fb88e5fbde904095d49"} Jul 10 17:03:28.991589 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:28.991540 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-h466f" podStartSLOduration=1.6443083349999998 podStartE2EDuration="5.991523101s" totalImagesPullingTime="4.347214766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:03:28.990257641 +0000 UTC m=+538.877549017" watchObservedRunningTime="2026-07-10 17:03:28.991523101 +0000 UTC m=+538.878814423" Jul 10 17:03:29.005825 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:29.005733 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" podStartSLOduration=0.754906137 podStartE2EDuration="6.005722619s" totalImagesPullingTime="5.250816482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:03:29.005595213 +0000 UTC m=+538.892886558" watchObservedRunningTime="2026-07-10 17:03:29.005722619 +0000 UTC m=+538.893013942" Jul 10 17:03:29.019221 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:29.019136 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-dt9wv" podStartSLOduration=0.774864287 podStartE2EDuration="5.01912246s" totalImagesPullingTime="4.244258173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:03:29.018424688 +0000 UTC m=+538.905716009" watchObservedRunningTime="2026-07-10 17:03:29.01912246 +0000 UTC m=+538.906413779" Jul 10 17:03:29.044699 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:29.044662 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-h466f"] Jul 10 17:03:30.988434 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:30.988372 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-h466f" podUID="5159719d-19c8-4316-b6d3-549f5b09ec52" containerName="authorino" containerID="cri-o://12b22d04befbdf166bc022f205bf6fc66698a2d6a3e6bdf1a527d1a59c0613d7" gracePeriod=30 Jul 10 17:03:31.236594 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:31.236570 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-h466f" Jul 10 17:03:31.276742 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:31.276643 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hbv9\" (UniqueName: \"kubernetes.io/projected/5159719d-19c8-4316-b6d3-549f5b09ec52-kube-api-access-9hbv9\") pod \"5159719d-19c8-4316-b6d3-549f5b09ec52\" (UID: \"5159719d-19c8-4316-b6d3-549f5b09ec52\") " Jul 10 17:03:31.278822 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:31.278792 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5159719d-19c8-4316-b6d3-549f5b09ec52-kube-api-access-9hbv9" (OuterVolumeSpecName: "kube-api-access-9hbv9") pod "5159719d-19c8-4316-b6d3-549f5b09ec52" (UID: "5159719d-19c8-4316-b6d3-549f5b09ec52"). InnerVolumeSpecName "kube-api-access-9hbv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:03:31.377852 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:31.377815 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hbv9\" (UniqueName: \"kubernetes.io/projected/5159719d-19c8-4316-b6d3-549f5b09ec52-kube-api-access-9hbv9\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:03:31.999609 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:31.999572 2567 generic.go:356] "Generic (PLEG): container finished" podID="5159719d-19c8-4316-b6d3-549f5b09ec52" containerID="12b22d04befbdf166bc022f205bf6fc66698a2d6a3e6bdf1a527d1a59c0613d7" exitCode=0 Jul 10 17:03:31.999985 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:31.999626 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-h466f" Jul 10 17:03:31.999985 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:31.999660 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-h466f" event={"ID":"5159719d-19c8-4316-b6d3-549f5b09ec52","Type":"ContainerDied","Data":"12b22d04befbdf166bc022f205bf6fc66698a2d6a3e6bdf1a527d1a59c0613d7"} Jul 10 17:03:31.999985 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:31.999701 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-h466f" event={"ID":"5159719d-19c8-4316-b6d3-549f5b09ec52","Type":"ContainerDied","Data":"335176671c3d78b7e5e974271edddd12f1cfa3655701b6cd4fb1628bb5846954"} Jul 10 17:03:31.999985 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:31.999717 2567 scope.go:117] "RemoveContainer" containerID="12b22d04befbdf166bc022f205bf6fc66698a2d6a3e6bdf1a527d1a59c0613d7" Jul 10 17:03:32.009319 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:32.009300 2567 scope.go:117] "RemoveContainer" containerID="12b22d04befbdf166bc022f205bf6fc66698a2d6a3e6bdf1a527d1a59c0613d7" Jul 10 17:03:32.009598 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:03:32.009579 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b22d04befbdf166bc022f205bf6fc66698a2d6a3e6bdf1a527d1a59c0613d7\": container with ID starting with 12b22d04befbdf166bc022f205bf6fc66698a2d6a3e6bdf1a527d1a59c0613d7 not found: ID does not exist" containerID="12b22d04befbdf166bc022f205bf6fc66698a2d6a3e6bdf1a527d1a59c0613d7" Jul 10 17:03:32.009683 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:32.009606 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b22d04befbdf166bc022f205bf6fc66698a2d6a3e6bdf1a527d1a59c0613d7"} err="failed to get container status \"12b22d04befbdf166bc022f205bf6fc66698a2d6a3e6bdf1a527d1a59c0613d7\": rpc error: code = NotFound desc = could not find container \"12b22d04befbdf166bc022f205bf6fc66698a2d6a3e6bdf1a527d1a59c0613d7\": container with ID starting with 12b22d04befbdf166bc022f205bf6fc66698a2d6a3e6bdf1a527d1a59c0613d7 not found: ID does not exist" Jul 10 17:03:32.022546 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:32.022523 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-h466f"] Jul 10 17:03:32.028251 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:32.028226 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-h466f"] Jul 10 17:03:32.681904 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:32.681865 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5159719d-19c8-4316-b6d3-549f5b09ec52" path="/var/lib/kubelet/pods/5159719d-19c8-4316-b6d3-549f5b09ec52/volumes" Jul 10 17:03:38.164543 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:38.164475 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-89qd9"] Jul 10 17:03:38.165134 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:38.164760 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" podUID="03e6ef32-7e75-4e08-acd3-17540daea739" containerName="limitador" containerID="cri-o://18bafdeab1f1a8254a3f597c5b99667172f1e72928b5efd8f7c1d5344d789695" gracePeriod=30 Jul 10 17:03:38.165625 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:38.165600 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" Jul 10 17:03:39.032380 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.032246 2567 generic.go:356] "Generic (PLEG): container finished" podID="03e6ef32-7e75-4e08-acd3-17540daea739" containerID="18bafdeab1f1a8254a3f597c5b99667172f1e72928b5efd8f7c1d5344d789695" exitCode=0 Jul 10 17:03:39.032380 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.032309 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" event={"ID":"03e6ef32-7e75-4e08-acd3-17540daea739","Type":"ContainerDied","Data":"18bafdeab1f1a8254a3f597c5b99667172f1e72928b5efd8f7c1d5344d789695"} Jul 10 17:03:39.123542 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.123511 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" Jul 10 17:03:39.248489 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.248398 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/03e6ef32-7e75-4e08-acd3-17540daea739-config-file\") pod \"03e6ef32-7e75-4e08-acd3-17540daea739\" (UID: \"03e6ef32-7e75-4e08-acd3-17540daea739\") " Jul 10 17:03:39.248928 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.248496 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z47g5\" (UniqueName: \"kubernetes.io/projected/03e6ef32-7e75-4e08-acd3-17540daea739-kube-api-access-z47g5\") pod \"03e6ef32-7e75-4e08-acd3-17540daea739\" (UID: \"03e6ef32-7e75-4e08-acd3-17540daea739\") " Jul 10 17:03:39.248928 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.248712 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e6ef32-7e75-4e08-acd3-17540daea739-config-file" (OuterVolumeSpecName: "config-file") pod "03e6ef32-7e75-4e08-acd3-17540daea739" (UID: "03e6ef32-7e75-4e08-acd3-17540daea739"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:03:39.250726 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.250689 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e6ef32-7e75-4e08-acd3-17540daea739-kube-api-access-z47g5" (OuterVolumeSpecName: "kube-api-access-z47g5") pod "03e6ef32-7e75-4e08-acd3-17540daea739" (UID: "03e6ef32-7e75-4e08-acd3-17540daea739"). InnerVolumeSpecName "kube-api-access-z47g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:03:39.349004 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.348966 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z47g5\" (UniqueName: \"kubernetes.io/projected/03e6ef32-7e75-4e08-acd3-17540daea739-kube-api-access-z47g5\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:03:39.349004 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.348996 2567 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/03e6ef32-7e75-4e08-acd3-17540daea739-config-file\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:03:39.643253 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.643171 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xh44p"] Jul 10 17:03:39.643518 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.643504 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5159719d-19c8-4316-b6d3-549f5b09ec52" containerName="authorino" Jul 10 17:03:39.643573 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.643520 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5159719d-19c8-4316-b6d3-549f5b09ec52" containerName="authorino" Jul 10 17:03:39.643573 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.643545 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e6ef32-7e75-4e08-acd3-17540daea739" containerName="limitador" Jul 10 17:03:39.643573 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.643551 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e6ef32-7e75-4e08-acd3-17540daea739" containerName="limitador" Jul 10 17:03:39.643658 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.643612 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5159719d-19c8-4316-b6d3-549f5b09ec52" containerName="authorino" Jul 10 17:03:39.643658 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.643622 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e6ef32-7e75-4e08-acd3-17540daea739" containerName="limitador" Jul 10 17:03:39.646524 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.646504 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xh44p" Jul 10 17:03:39.649358 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.649340 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 17:03:39.649447 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.649370 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 17:03:39.649534 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.649515 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jzdqm\"" Jul 10 17:03:39.650030 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.650014 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 17:03:39.657458 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.657439 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xh44p"] Jul 10 17:03:39.750622 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.750595 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb9bn\" (UniqueName: \"kubernetes.io/projected/160440f6-fcf4-4837-a170-cc0f4089df6c-kube-api-access-cb9bn\") pod \"postgres-69fb4594cf-xh44p\" (UID: \"160440f6-fcf4-4837-a170-cc0f4089df6c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xh44p" Jul 10 17:03:39.750802 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.750646 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/160440f6-fcf4-4837-a170-cc0f4089df6c-data\") pod \"postgres-69fb4594cf-xh44p\" (UID: \"160440f6-fcf4-4837-a170-cc0f4089df6c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xh44p" Jul 10 17:03:39.851554 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.851519 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb9bn\" (UniqueName: \"kubernetes.io/projected/160440f6-fcf4-4837-a170-cc0f4089df6c-kube-api-access-cb9bn\") pod \"postgres-69fb4594cf-xh44p\" (UID: \"160440f6-fcf4-4837-a170-cc0f4089df6c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xh44p" Jul 10 17:03:39.851692 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.851572 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/160440f6-fcf4-4837-a170-cc0f4089df6c-data\") pod \"postgres-69fb4594cf-xh44p\" (UID: \"160440f6-fcf4-4837-a170-cc0f4089df6c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xh44p" Jul 10 17:03:39.851872 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.851858 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/160440f6-fcf4-4837-a170-cc0f4089df6c-data\") pod \"postgres-69fb4594cf-xh44p\" (UID: \"160440f6-fcf4-4837-a170-cc0f4089df6c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xh44p" Jul 10 17:03:39.864535 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.864506 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb9bn\" (UniqueName: \"kubernetes.io/projected/160440f6-fcf4-4837-a170-cc0f4089df6c-kube-api-access-cb9bn\") pod \"postgres-69fb4594cf-xh44p\" (UID: \"160440f6-fcf4-4837-a170-cc0f4089df6c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xh44p" Jul 10 17:03:39.956660 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:39.956633 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xh44p" Jul 10 17:03:40.039426 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:40.039203 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" Jul 10 17:03:40.039426 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:40.039238 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-89qd9" event={"ID":"03e6ef32-7e75-4e08-acd3-17540daea739","Type":"ContainerDied","Data":"e2cdb9dfe6c97a4fe48ff29bba2e7b1808f562e03312b6e64b8a0a51a766d8a6"} Jul 10 17:03:40.039426 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:40.039335 2567 scope.go:117] "RemoveContainer" containerID="18bafdeab1f1a8254a3f597c5b99667172f1e72928b5efd8f7c1d5344d789695" Jul 10 17:03:40.069966 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:03:40.069930 2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e6ef32_7e75_4e08_acd3_17540daea739.slice/crio-e2cdb9dfe6c97a4fe48ff29bba2e7b1808f562e03312b6e64b8a0a51a766d8a6\": RecentStats: unable to find data in memory cache]" Jul 10 17:03:40.070103 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:03:40.069930 2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e6ef32_7e75_4e08_acd3_17540daea739.slice/crio-e2cdb9dfe6c97a4fe48ff29bba2e7b1808f562e03312b6e64b8a0a51a766d8a6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e6ef32_7e75_4e08_acd3_17540daea739.slice\": RecentStats: unable to find data in memory cache]" Jul 10 17:03:40.072911 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:40.072885 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-89qd9"] Jul 10 17:03:40.077889 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:40.077865 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-89qd9"] Jul 10 17:03:40.092166 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:40.092146 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xh44p"] Jul 10 17:03:40.094172 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:03:40.094153 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160440f6_fcf4_4837_a170_cc0f4089df6c.slice/crio-4662846e03d8ca8abce9e24e1d7c3a2ca32211c317d5ffe90d57f3ebabf6c106 WatchSource:0}: Error finding container 4662846e03d8ca8abce9e24e1d7c3a2ca32211c317d5ffe90d57f3ebabf6c106: Status 404 returned error can't find the container with id 4662846e03d8ca8abce9e24e1d7c3a2ca32211c317d5ffe90d57f3ebabf6c106 Jul 10 17:03:40.682373 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:40.682343 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03e6ef32-7e75-4e08-acd3-17540daea739" path="/var/lib/kubelet/pods/03e6ef32-7e75-4e08-acd3-17540daea739/volumes" Jul 10 17:03:41.047625 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:41.047569 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xh44p" event={"ID":"160440f6-fcf4-4837-a170-cc0f4089df6c","Type":"ContainerStarted","Data":"4662846e03d8ca8abce9e24e1d7c3a2ca32211c317d5ffe90d57f3ebabf6c106"} Jul 10 17:03:45.073204 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:45.072622 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xh44p" event={"ID":"160440f6-fcf4-4837-a170-cc0f4089df6c","Type":"ContainerStarted","Data":"3990a67029d9926530892526567b2ddd4b442fa9b9c7d19351ae17c58add9749"} Jul 10 17:03:45.073657 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:45.073200 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xh44p" Jul 10 17:03:45.089817 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:45.089758 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xh44p" podStartSLOduration=1.363681222 podStartE2EDuration="6.089739289s" totalImagesPullingTime="4.726058067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:03:45.089044416 +0000 UTC m=+554.976335736" watchObservedRunningTime="2026-07-10 17:03:45.089739289 +0000 UTC m=+554.977030609" Jul 10 17:03:52.112039 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:52.112008 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xh44p" Jul 10 17:03:52.661606 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:52.661569 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5hqzw"] Jul 10 17:03:52.663740 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:52.663724 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5hqzw" Jul 10 17:03:52.673390 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:52.673365 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5hqzw"] Jul 10 17:03:52.772212 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:52.772171 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjsvt\" (UniqueName: \"kubernetes.io/projected/87d5c021-8134-4fdd-a7d1-c20a8f0f0b57-kube-api-access-wjsvt\") pod \"authorino-5d4b67c446-5hqzw\" (UID: \"87d5c021-8134-4fdd-a7d1-c20a8f0f0b57\") " pod="kuadrant-system/authorino-5d4b67c446-5hqzw" Jul 10 17:03:52.872631 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:52.872595 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsvt\" (UniqueName: \"kubernetes.io/projected/87d5c021-8134-4fdd-a7d1-c20a8f0f0b57-kube-api-access-wjsvt\") pod \"authorino-5d4b67c446-5hqzw\" (UID: \"87d5c021-8134-4fdd-a7d1-c20a8f0f0b57\") " pod="kuadrant-system/authorino-5d4b67c446-5hqzw" Jul 10 17:03:52.883401 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:52.883373 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjsvt\" (UniqueName: \"kubernetes.io/projected/87d5c021-8134-4fdd-a7d1-c20a8f0f0b57-kube-api-access-wjsvt\") pod \"authorino-5d4b67c446-5hqzw\" (UID: \"87d5c021-8134-4fdd-a7d1-c20a8f0f0b57\") " pod="kuadrant-system/authorino-5d4b67c446-5hqzw" Jul 10 17:03:52.912732 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:52.912646 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5hqzw"] Jul 10 17:03:52.912917 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:52.912902 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5hqzw" Jul 10 17:03:52.951833 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:52.951791 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c89fb46c-4nxqj"] Jul 10 17:03:52.954797 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:52.954439 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c89fb46c-4nxqj" Jul 10 17:03:52.964791 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:52.964727 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c89fb46c-4nxqj"] Jul 10 17:03:53.053002 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.052978 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5hqzw"] Jul 10 17:03:53.055092 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:03:53.055062 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87d5c021_8134_4fdd_a7d1_c20a8f0f0b57.slice/crio-ec121b8e94469750fccba81f9d13b3de20450b60904df84f37a70bf338b990bf WatchSource:0}: Error finding container ec121b8e94469750fccba81f9d13b3de20450b60904df84f37a70bf338b990bf: Status 404 returned error can't find the container with id ec121b8e94469750fccba81f9d13b3de20450b60904df84f37a70bf338b990bf Jul 10 17:03:53.074294 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.074239 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdqhs\" (UniqueName: \"kubernetes.io/projected/63ace9ba-9dfb-4914-be0e-379bf92b62e6-kube-api-access-pdqhs\") pod \"authorino-6c89fb46c-4nxqj\" (UID: \"63ace9ba-9dfb-4914-be0e-379bf92b62e6\") " pod="kuadrant-system/authorino-6c89fb46c-4nxqj" Jul 10 17:03:53.114334 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.114297 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5hqzw" event={"ID":"87d5c021-8134-4fdd-a7d1-c20a8f0f0b57","Type":"ContainerStarted","Data":"ec121b8e94469750fccba81f9d13b3de20450b60904df84f37a70bf338b990bf"} Jul 10 17:03:53.175122 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.175019 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdqhs\" (UniqueName: \"kubernetes.io/projected/63ace9ba-9dfb-4914-be0e-379bf92b62e6-kube-api-access-pdqhs\") pod \"authorino-6c89fb46c-4nxqj\" (UID: \"63ace9ba-9dfb-4914-be0e-379bf92b62e6\") " pod="kuadrant-system/authorino-6c89fb46c-4nxqj" Jul 10 17:03:53.183479 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.183449 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdqhs\" (UniqueName: \"kubernetes.io/projected/63ace9ba-9dfb-4914-be0e-379bf92b62e6-kube-api-access-pdqhs\") pod \"authorino-6c89fb46c-4nxqj\" (UID: \"63ace9ba-9dfb-4914-be0e-379bf92b62e6\") " pod="kuadrant-system/authorino-6c89fb46c-4nxqj" Jul 10 17:03:53.205244 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.205210 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c89fb46c-4nxqj"] Jul 10 17:03:53.205480 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.205459 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c89fb46c-4nxqj" Jul 10 17:03:53.234663 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.234611 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-564b958767-prqt8"] Jul 10 17:03:53.237470 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.237432 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-564b958767-prqt8" Jul 10 17:03:53.240004 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.239980 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 17:03:53.245014 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.244987 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-564b958767-prqt8"] Jul 10 17:03:53.346594 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.346571 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c89fb46c-4nxqj"] Jul 10 17:03:53.348638 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:03:53.348613 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63ace9ba_9dfb_4914_be0e_379bf92b62e6.slice/crio-64b26836d23afb734b7527d564a6832d826950394059cf0d14e7828139dc95c9 WatchSource:0}: Error finding container 64b26836d23afb734b7527d564a6832d826950394059cf0d14e7828139dc95c9: Status 404 returned error can't find the container with id 64b26836d23afb734b7527d564a6832d826950394059cf0d14e7828139dc95c9 Jul 10 17:03:53.376560 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.376534 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/233ed9ef-cdae-4c66-88cc-049bd062935a-tls-cert\") pod \"authorino-564b958767-prqt8\" (UID: \"233ed9ef-cdae-4c66-88cc-049bd062935a\") " pod="kuadrant-system/authorino-564b958767-prqt8" Jul 10 17:03:53.376662 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.376568 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djp4j\" (UniqueName: \"kubernetes.io/projected/233ed9ef-cdae-4c66-88cc-049bd062935a-kube-api-access-djp4j\") pod \"authorino-564b958767-prqt8\" (UID: \"233ed9ef-cdae-4c66-88cc-049bd062935a\") " pod="kuadrant-system/authorino-564b958767-prqt8" Jul 10 17:03:53.477439 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.477409 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/233ed9ef-cdae-4c66-88cc-049bd062935a-tls-cert\") pod \"authorino-564b958767-prqt8\" (UID: \"233ed9ef-cdae-4c66-88cc-049bd062935a\") " pod="kuadrant-system/authorino-564b958767-prqt8" Jul 10 17:03:53.477647 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.477450 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djp4j\" (UniqueName: \"kubernetes.io/projected/233ed9ef-cdae-4c66-88cc-049bd062935a-kube-api-access-djp4j\") pod \"authorino-564b958767-prqt8\" (UID: \"233ed9ef-cdae-4c66-88cc-049bd062935a\") " pod="kuadrant-system/authorino-564b958767-prqt8" Jul 10 17:03:53.481416 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.481104 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/233ed9ef-cdae-4c66-88cc-049bd062935a-tls-cert\") pod \"authorino-564b958767-prqt8\" (UID: \"233ed9ef-cdae-4c66-88cc-049bd062935a\") " pod="kuadrant-system/authorino-564b958767-prqt8" Jul 10 17:03:53.486902 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.486871 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djp4j\" (UniqueName: \"kubernetes.io/projected/233ed9ef-cdae-4c66-88cc-049bd062935a-kube-api-access-djp4j\") pod \"authorino-564b958767-prqt8\" (UID: \"233ed9ef-cdae-4c66-88cc-049bd062935a\") " pod="kuadrant-system/authorino-564b958767-prqt8" Jul 10 17:03:53.549476 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.549432 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-564b958767-prqt8" Jul 10 17:03:53.686813 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:53.686782 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-564b958767-prqt8"] Jul 10 17:03:53.689186 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:03:53.689148 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod233ed9ef_cdae_4c66_88cc_049bd062935a.slice/crio-57f178ffe34353c58c83975fa1dc7f2af1952bf526bf047541748e605675331b WatchSource:0}: Error finding container 57f178ffe34353c58c83975fa1dc7f2af1952bf526bf047541748e605675331b: Status 404 returned error can't find the container with id 57f178ffe34353c58c83975fa1dc7f2af1952bf526bf047541748e605675331b Jul 10 17:03:54.120878 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.120838 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5hqzw" event={"ID":"87d5c021-8134-4fdd-a7d1-c20a8f0f0b57","Type":"ContainerStarted","Data":"f9f709018b458cf22adb67da4a0b947b904a030d6576a48eec8ab85249d4a79c"} Jul 10 17:03:54.121442 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.120899 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-5hqzw" podUID="87d5c021-8134-4fdd-a7d1-c20a8f0f0b57" containerName="authorino" containerID="cri-o://f9f709018b458cf22adb67da4a0b947b904a030d6576a48eec8ab85249d4a79c" gracePeriod=30 Jul 10 17:03:54.122795 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.122766 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c89fb46c-4nxqj" event={"ID":"63ace9ba-9dfb-4914-be0e-379bf92b62e6","Type":"ContainerStarted","Data":"d008bc75bafa414f866450bbd42463413902063377b7f5df260fa40469e87015"} Jul 10 17:03:54.122946 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.122809 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c89fb46c-4nxqj" event={"ID":"63ace9ba-9dfb-4914-be0e-379bf92b62e6","Type":"ContainerStarted","Data":"64b26836d23afb734b7527d564a6832d826950394059cf0d14e7828139dc95c9"} Jul 10 17:03:54.122946 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.122769 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6c89fb46c-4nxqj" podUID="63ace9ba-9dfb-4914-be0e-379bf92b62e6" containerName="authorino" containerID="cri-o://d008bc75bafa414f866450bbd42463413902063377b7f5df260fa40469e87015" gracePeriod=30 Jul 10 17:03:54.124920 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.124889 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-564b958767-prqt8" event={"ID":"233ed9ef-cdae-4c66-88cc-049bd062935a","Type":"ContainerStarted","Data":"8a77beeca10d61df056ec289484c086d6ea7f1b2ebc28a17eae23278f12964cb"} Jul 10 17:03:54.125046 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.124934 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-564b958767-prqt8" event={"ID":"233ed9ef-cdae-4c66-88cc-049bd062935a","Type":"ContainerStarted","Data":"57f178ffe34353c58c83975fa1dc7f2af1952bf526bf047541748e605675331b"} Jul 10 17:03:54.137098 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.137038 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-5hqzw" podStartSLOduration=1.833805653 podStartE2EDuration="2.13701882s" totalImagesPullingTime="303.213167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:03:54.136729811 +0000 UTC m=+564.024021133" watchObservedRunningTime="2026-07-10 17:03:54.13701882 +0000 UTC m=+564.024310332" Jul 10 17:03:54.155061 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.154996 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-564b958767-prqt8" podStartSLOduration=0.811567782 podStartE2EDuration="1.15497364s" totalImagesPullingTime="343.405858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:03:54.153559471 +0000 UTC m=+564.040850785" watchObservedRunningTime="2026-07-10 17:03:54.15497364 +0000 UTC m=+564.042264961" Jul 10 17:03:54.170960 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.170889 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c89fb46c-4nxqj" podStartSLOduration=1.810891381 podStartE2EDuration="2.170862846s" totalImagesPullingTime="359.971465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:03:54.1693025 +0000 UTC m=+564.056593872" watchObservedRunningTime="2026-07-10 17:03:54.170862846 +0000 UTC m=+564.058154167" Jul 10 17:03:54.187558 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.187369 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dt9wv"] Jul 10 17:03:54.188465 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.187885 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-dt9wv" podUID="322d75e8-4004-4f21-983e-fa05d2833ece" containerName="authorino" containerID="cri-o://089a6001091037b28ed6b102e07d7f04746ba01c74264fb88e5fbde904095d49" gracePeriod=30 Jul 10 17:03:54.468966 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.468930 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5hqzw" Jul 10 17:03:54.514728 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.514704 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c89fb46c-4nxqj" Jul 10 17:03:54.520544 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.520516 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dt9wv" Jul 10 17:03:54.588486 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.588389 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjsvt\" (UniqueName: \"kubernetes.io/projected/87d5c021-8134-4fdd-a7d1-c20a8f0f0b57-kube-api-access-wjsvt\") pod \"87d5c021-8134-4fdd-a7d1-c20a8f0f0b57\" (UID: \"87d5c021-8134-4fdd-a7d1-c20a8f0f0b57\") " Jul 10 17:03:54.590627 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.590602 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d5c021-8134-4fdd-a7d1-c20a8f0f0b57-kube-api-access-wjsvt" (OuterVolumeSpecName: "kube-api-access-wjsvt") pod "87d5c021-8134-4fdd-a7d1-c20a8f0f0b57" (UID: "87d5c021-8134-4fdd-a7d1-c20a8f0f0b57"). InnerVolumeSpecName "kube-api-access-wjsvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:03:54.689483 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.689443 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdqhs\" (UniqueName: \"kubernetes.io/projected/63ace9ba-9dfb-4914-be0e-379bf92b62e6-kube-api-access-pdqhs\") pod \"63ace9ba-9dfb-4914-be0e-379bf92b62e6\" (UID: \"63ace9ba-9dfb-4914-be0e-379bf92b62e6\") " Jul 10 17:03:54.689634 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.689528 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwcc7\" (UniqueName: \"kubernetes.io/projected/322d75e8-4004-4f21-983e-fa05d2833ece-kube-api-access-kwcc7\") pod \"322d75e8-4004-4f21-983e-fa05d2833ece\" (UID: \"322d75e8-4004-4f21-983e-fa05d2833ece\") " Jul 10 17:03:54.689674 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.689665 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjsvt\" (UniqueName: \"kubernetes.io/projected/87d5c021-8134-4fdd-a7d1-c20a8f0f0b57-kube-api-access-wjsvt\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:03:54.692031 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.691995 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ace9ba-9dfb-4914-be0e-379bf92b62e6-kube-api-access-pdqhs" (OuterVolumeSpecName: "kube-api-access-pdqhs") pod "63ace9ba-9dfb-4914-be0e-379bf92b62e6" (UID: "63ace9ba-9dfb-4914-be0e-379bf92b62e6"). InnerVolumeSpecName "kube-api-access-pdqhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:03:54.692143 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.692041 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322d75e8-4004-4f21-983e-fa05d2833ece-kube-api-access-kwcc7" (OuterVolumeSpecName: "kube-api-access-kwcc7") pod "322d75e8-4004-4f21-983e-fa05d2833ece" (UID: "322d75e8-4004-4f21-983e-fa05d2833ece"). InnerVolumeSpecName "kube-api-access-kwcc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:03:54.790435 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.790397 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdqhs\" (UniqueName: \"kubernetes.io/projected/63ace9ba-9dfb-4914-be0e-379bf92b62e6-kube-api-access-pdqhs\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:03:54.790435 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:54.790430 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwcc7\" (UniqueName: \"kubernetes.io/projected/322d75e8-4004-4f21-983e-fa05d2833ece-kube-api-access-kwcc7\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:03:55.131501 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.131459 2567 generic.go:356] "Generic (PLEG): container finished" podID="87d5c021-8134-4fdd-a7d1-c20a8f0f0b57" containerID="f9f709018b458cf22adb67da4a0b947b904a030d6576a48eec8ab85249d4a79c" exitCode=0 Jul 10 17:03:55.131501 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.131503 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5hqzw" Jul 10 17:03:55.132088 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.131540 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5hqzw" event={"ID":"87d5c021-8134-4fdd-a7d1-c20a8f0f0b57","Type":"ContainerDied","Data":"f9f709018b458cf22adb67da4a0b947b904a030d6576a48eec8ab85249d4a79c"} Jul 10 17:03:55.132088 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.131591 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5hqzw" event={"ID":"87d5c021-8134-4fdd-a7d1-c20a8f0f0b57","Type":"ContainerDied","Data":"ec121b8e94469750fccba81f9d13b3de20450b60904df84f37a70bf338b990bf"} Jul 10 17:03:55.132088 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.131616 2567 scope.go:117] "RemoveContainer" containerID="f9f709018b458cf22adb67da4a0b947b904a030d6576a48eec8ab85249d4a79c" Jul 10 17:03:55.133100 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.133076 2567 generic.go:356] "Generic (PLEG): container finished" podID="63ace9ba-9dfb-4914-be0e-379bf92b62e6" containerID="d008bc75bafa414f866450bbd42463413902063377b7f5df260fa40469e87015" exitCode=0 Jul 10 17:03:55.133193 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.133122 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c89fb46c-4nxqj" Jul 10 17:03:55.133193 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.133164 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c89fb46c-4nxqj" event={"ID":"63ace9ba-9dfb-4914-be0e-379bf92b62e6","Type":"ContainerDied","Data":"d008bc75bafa414f866450bbd42463413902063377b7f5df260fa40469e87015"} Jul 10 17:03:55.133496 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.133205 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c89fb46c-4nxqj" event={"ID":"63ace9ba-9dfb-4914-be0e-379bf92b62e6","Type":"ContainerDied","Data":"64b26836d23afb734b7527d564a6832d826950394059cf0d14e7828139dc95c9"} Jul 10 17:03:55.142072 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.141994 2567 generic.go:356] "Generic (PLEG): container finished" podID="322d75e8-4004-4f21-983e-fa05d2833ece" containerID="089a6001091037b28ed6b102e07d7f04746ba01c74264fb88e5fbde904095d49" exitCode=0 Jul 10 17:03:55.142208 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.142121 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dt9wv" Jul 10 17:03:55.142208 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.142192 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dt9wv" event={"ID":"322d75e8-4004-4f21-983e-fa05d2833ece","Type":"ContainerDied","Data":"089a6001091037b28ed6b102e07d7f04746ba01c74264fb88e5fbde904095d49"} Jul 10 17:03:55.142347 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.142225 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dt9wv" event={"ID":"322d75e8-4004-4f21-983e-fa05d2833ece","Type":"ContainerDied","Data":"e6b050a84eae6130097570cd3c1b9ad9d7910b8b679fbc1ca586245d376a662a"} Jul 10 17:03:55.146850 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.146830 2567 scope.go:117] "RemoveContainer" containerID="f9f709018b458cf22adb67da4a0b947b904a030d6576a48eec8ab85249d4a79c" Jul 10 17:03:55.147122 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:03:55.147103 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f709018b458cf22adb67da4a0b947b904a030d6576a48eec8ab85249d4a79c\": container with ID starting with f9f709018b458cf22adb67da4a0b947b904a030d6576a48eec8ab85249d4a79c not found: ID does not exist" containerID="f9f709018b458cf22adb67da4a0b947b904a030d6576a48eec8ab85249d4a79c" Jul 10 17:03:55.147213 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.147126 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f709018b458cf22adb67da4a0b947b904a030d6576a48eec8ab85249d4a79c"} err="failed to get container status \"f9f709018b458cf22adb67da4a0b947b904a030d6576a48eec8ab85249d4a79c\": rpc error: code = NotFound desc = could not find container \"f9f709018b458cf22adb67da4a0b947b904a030d6576a48eec8ab85249d4a79c\": container with ID starting with f9f709018b458cf22adb67da4a0b947b904a030d6576a48eec8ab85249d4a79c not found: ID does not exist" Jul 10 17:03:55.147213 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.147142 2567 scope.go:117] "RemoveContainer" containerID="d008bc75bafa414f866450bbd42463413902063377b7f5df260fa40469e87015" Jul 10 17:03:55.155299 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.155260 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5hqzw"] Jul 10 17:03:55.158315 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.158265 2567 scope.go:117] "RemoveContainer" containerID="d008bc75bafa414f866450bbd42463413902063377b7f5df260fa40469e87015" Jul 10 17:03:55.158628 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:03:55.158605 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d008bc75bafa414f866450bbd42463413902063377b7f5df260fa40469e87015\": container with ID starting with d008bc75bafa414f866450bbd42463413902063377b7f5df260fa40469e87015 not found: ID does not exist" containerID="d008bc75bafa414f866450bbd42463413902063377b7f5df260fa40469e87015" Jul 10 17:03:55.158718 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.158632 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d008bc75bafa414f866450bbd42463413902063377b7f5df260fa40469e87015"} err="failed to get container status \"d008bc75bafa414f866450bbd42463413902063377b7f5df260fa40469e87015\": rpc error: code = NotFound desc = could not find container \"d008bc75bafa414f866450bbd42463413902063377b7f5df260fa40469e87015\": container with ID starting with d008bc75bafa414f866450bbd42463413902063377b7f5df260fa40469e87015 not found: ID does not exist" Jul 10 17:03:55.158718 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.158649 2567 scope.go:117] "RemoveContainer" containerID="089a6001091037b28ed6b102e07d7f04746ba01c74264fb88e5fbde904095d49" Jul 10 17:03:55.159044 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.159025 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5hqzw"] Jul 10 17:03:55.166242 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.166226 2567 scope.go:117] "RemoveContainer" containerID="089a6001091037b28ed6b102e07d7f04746ba01c74264fb88e5fbde904095d49" Jul 10 17:03:55.166509 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:03:55.166492 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089a6001091037b28ed6b102e07d7f04746ba01c74264fb88e5fbde904095d49\": container with ID starting with 089a6001091037b28ed6b102e07d7f04746ba01c74264fb88e5fbde904095d49 not found: ID does not exist" containerID="089a6001091037b28ed6b102e07d7f04746ba01c74264fb88e5fbde904095d49" Jul 10 17:03:55.166553 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.166517 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089a6001091037b28ed6b102e07d7f04746ba01c74264fb88e5fbde904095d49"} err="failed to get container status \"089a6001091037b28ed6b102e07d7f04746ba01c74264fb88e5fbde904095d49\": rpc error: code = NotFound desc = could not find container \"089a6001091037b28ed6b102e07d7f04746ba01c74264fb88e5fbde904095d49\": container with ID starting with 089a6001091037b28ed6b102e07d7f04746ba01c74264fb88e5fbde904095d49 not found: ID does not exist" Jul 10 17:03:55.171018 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.170997 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dt9wv"] Jul 10 17:03:55.175409 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.175390 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dt9wv"] Jul 10 17:03:55.184568 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.184546 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c89fb46c-4nxqj"] Jul 10 17:03:55.188931 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:55.188913 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c89fb46c-4nxqj"] Jul 10 17:03:56.681228 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:56.681195 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322d75e8-4004-4f21-983e-fa05d2833ece" path="/var/lib/kubelet/pods/322d75e8-4004-4f21-983e-fa05d2833ece/volumes" Jul 10 17:03:56.681618 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:56.681536 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63ace9ba-9dfb-4914-be0e-379bf92b62e6" path="/var/lib/kubelet/pods/63ace9ba-9dfb-4914-be0e-379bf92b62e6/volumes" Jul 10 17:03:56.681817 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:56.681804 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87d5c021-8134-4fdd-a7d1-c20a8f0f0b57" path="/var/lib/kubelet/pods/87d5c021-8134-4fdd-a7d1-c20a8f0f0b57/volumes" Jul 10 17:03:58.965994 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:58.965964 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b46985b8-8jcl7"] Jul 10 17:03:58.966381 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:58.966309 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d5c021-8134-4fdd-a7d1-c20a8f0f0b57" containerName="authorino" Jul 10 17:03:58.966381 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:58.966321 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d5c021-8134-4fdd-a7d1-c20a8f0f0b57" containerName="authorino" Jul 10 17:03:58.966381 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:58.966333 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322d75e8-4004-4f21-983e-fa05d2833ece" containerName="authorino" Jul 10 17:03:58.966381 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:58.966338 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="322d75e8-4004-4f21-983e-fa05d2833ece" containerName="authorino" Jul 10 17:03:58.966381 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:58.966346 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63ace9ba-9dfb-4914-be0e-379bf92b62e6" containerName="authorino" Jul 10 17:03:58.966381 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:58.966351 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="63ace9ba-9dfb-4914-be0e-379bf92b62e6" containerName="authorino" Jul 10 17:03:58.966594 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:58.966413 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="322d75e8-4004-4f21-983e-fa05d2833ece" containerName="authorino" Jul 10 17:03:58.966594 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:58.966420 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d5c021-8134-4fdd-a7d1-c20a8f0f0b57" containerName="authorino" Jul 10 17:03:58.966594 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:58.966427 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="63ace9ba-9dfb-4914-be0e-379bf92b62e6" containerName="authorino" Jul 10 17:03:59.141971 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.141923 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b46985b8-8jcl7"] Jul 10 17:03:59.142143 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.141985 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b46985b8-8jcl7" Jul 10 17:03:59.143307 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.143266 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c76c8d8b8-fvbbn"] Jul 10 17:03:59.145148 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.145107 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dr5rg\"" Jul 10 17:03:59.145148 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.145134 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 17:03:59.145382 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.145110 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 17:03:59.173786 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.173762 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c76c8d8b8-fvbbn"] Jul 10 17:03:59.173929 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.173855 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" Jul 10 17:03:59.232540 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.232476 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swngq\" (UniqueName: \"kubernetes.io/projected/24f1b9d3-c454-4e09-8707-1505b56970a1-kube-api-access-swngq\") pod \"maas-controller-5b46985b8-8jcl7\" (UID: \"24f1b9d3-c454-4e09-8707-1505b56970a1\") " pod="opendatahub/maas-controller-5b46985b8-8jcl7" Jul 10 17:03:59.232666 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.232544 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24f1b9d3-c454-4e09-8707-1505b56970a1-webhook-cert\") pod \"maas-controller-5b46985b8-8jcl7\" (UID: \"24f1b9d3-c454-4e09-8707-1505b56970a1\") " pod="opendatahub/maas-controller-5b46985b8-8jcl7" Jul 10 17:03:59.333816 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.333785 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swngq\" (UniqueName: \"kubernetes.io/projected/24f1b9d3-c454-4e09-8707-1505b56970a1-kube-api-access-swngq\") pod \"maas-controller-5b46985b8-8jcl7\" (UID: \"24f1b9d3-c454-4e09-8707-1505b56970a1\") " pod="opendatahub/maas-controller-5b46985b8-8jcl7" Jul 10 17:03:59.333928 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.333836 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4d91000-362c-4c24-a6a2-5d44a7e91ea1-webhook-cert\") pod \"maas-controller-6c76c8d8b8-fvbbn\" (UID: \"b4d91000-362c-4c24-a6a2-5d44a7e91ea1\") " pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" Jul 10 17:03:59.333928 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.333874 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfndw\" (UniqueName: \"kubernetes.io/projected/b4d91000-362c-4c24-a6a2-5d44a7e91ea1-kube-api-access-nfndw\") pod \"maas-controller-6c76c8d8b8-fvbbn\" (UID: \"b4d91000-362c-4c24-a6a2-5d44a7e91ea1\") " pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" Jul 10 17:03:59.333928 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.333892 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24f1b9d3-c454-4e09-8707-1505b56970a1-webhook-cert\") pod \"maas-controller-5b46985b8-8jcl7\" (UID: \"24f1b9d3-c454-4e09-8707-1505b56970a1\") " pod="opendatahub/maas-controller-5b46985b8-8jcl7" Jul 10 17:03:59.336383 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.336361 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24f1b9d3-c454-4e09-8707-1505b56970a1-webhook-cert\") pod \"maas-controller-5b46985b8-8jcl7\" (UID: \"24f1b9d3-c454-4e09-8707-1505b56970a1\") " pod="opendatahub/maas-controller-5b46985b8-8jcl7" Jul 10 17:03:59.342478 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.342453 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swngq\" (UniqueName: \"kubernetes.io/projected/24f1b9d3-c454-4e09-8707-1505b56970a1-kube-api-access-swngq\") pod \"maas-controller-5b46985b8-8jcl7\" (UID: \"24f1b9d3-c454-4e09-8707-1505b56970a1\") " pod="opendatahub/maas-controller-5b46985b8-8jcl7" Jul 10 17:03:59.435189 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.435155 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4d91000-362c-4c24-a6a2-5d44a7e91ea1-webhook-cert\") pod \"maas-controller-6c76c8d8b8-fvbbn\" (UID: \"b4d91000-362c-4c24-a6a2-5d44a7e91ea1\") " pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" Jul 10 17:03:59.435484 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.435461 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfndw\" (UniqueName: \"kubernetes.io/projected/b4d91000-362c-4c24-a6a2-5d44a7e91ea1-kube-api-access-nfndw\") pod \"maas-controller-6c76c8d8b8-fvbbn\" (UID: \"b4d91000-362c-4c24-a6a2-5d44a7e91ea1\") " pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" Jul 10 17:03:59.438991 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.438920 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4d91000-362c-4c24-a6a2-5d44a7e91ea1-webhook-cert\") pod \"maas-controller-6c76c8d8b8-fvbbn\" (UID: \"b4d91000-362c-4c24-a6a2-5d44a7e91ea1\") " pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" Jul 10 17:03:59.447193 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.447166 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfndw\" (UniqueName: \"kubernetes.io/projected/b4d91000-362c-4c24-a6a2-5d44a7e91ea1-kube-api-access-nfndw\") pod \"maas-controller-6c76c8d8b8-fvbbn\" (UID: \"b4d91000-362c-4c24-a6a2-5d44a7e91ea1\") " pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" Jul 10 17:03:59.452041 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.452022 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b46985b8-8jcl7" Jul 10 17:03:59.483397 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.483346 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" Jul 10 17:03:59.637635 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.637605 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b46985b8-8jcl7"] Jul 10 17:03:59.638935 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:03:59.638893 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f1b9d3_c454_4e09_8707_1505b56970a1.slice/crio-1c5ee770ea227336f9e359ac8473a0776ec647e8995ab0ac26dfad3b8d9fcd64 WatchSource:0}: Error finding container 1c5ee770ea227336f9e359ac8473a0776ec647e8995ab0ac26dfad3b8d9fcd64: Status 404 returned error can't find the container with id 1c5ee770ea227336f9e359ac8473a0776ec647e8995ab0ac26dfad3b8d9fcd64 Jul 10 17:03:59.661936 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:03:59.661911 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c76c8d8b8-fvbbn"] Jul 10 17:03:59.663505 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:03:59.663486 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d91000_362c_4c24_a6a2_5d44a7e91ea1.slice/crio-682a9f5b41fe39da9b32a741b9f06396cfc109a04627eb5fe475e216b3185a92 WatchSource:0}: Error finding container 682a9f5b41fe39da9b32a741b9f06396cfc109a04627eb5fe475e216b3185a92: Status 404 returned error can't find the container with id 682a9f5b41fe39da9b32a741b9f06396cfc109a04627eb5fe475e216b3185a92 Jul 10 17:04:00.169152 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:00.169117 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b46985b8-8jcl7" event={"ID":"24f1b9d3-c454-4e09-8707-1505b56970a1","Type":"ContainerStarted","Data":"1c5ee770ea227336f9e359ac8473a0776ec647e8995ab0ac26dfad3b8d9fcd64"} Jul 10 17:04:00.170294 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:00.170249 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" event={"ID":"b4d91000-362c-4c24-a6a2-5d44a7e91ea1","Type":"ContainerStarted","Data":"682a9f5b41fe39da9b32a741b9f06396cfc109a04627eb5fe475e216b3185a92"} Jul 10 17:04:03.188261 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:03.188220 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b46985b8-8jcl7" event={"ID":"24f1b9d3-c454-4e09-8707-1505b56970a1","Type":"ContainerStarted","Data":"2b1998c5829cf35db745b8ca9fa80f6f2e71b73e7f65c9a4b6f5e5f593ea0dfa"} Jul 10 17:04:03.188898 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:03.188306 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b46985b8-8jcl7" Jul 10 17:04:03.190340 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:03.190313 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" event={"ID":"b4d91000-362c-4c24-a6a2-5d44a7e91ea1","Type":"ContainerStarted","Data":"fde3f5f60880e57365435ed81621a7bf2389d98b684ead4bfde55f917fe6d333"} Jul 10 17:04:03.190453 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:03.190371 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" Jul 10 17:04:03.213001 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:03.212949 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b46985b8-8jcl7" podStartSLOduration=2.5369893599999997 podStartE2EDuration="5.212934826s" totalImagesPullingTime="2.675945466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:03:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:04:03.211479705 +0000 UTC m=+573.098771027" watchObservedRunningTime="2026-07-10 17:04:03.212934826 +0000 UTC m=+573.100226145" Jul 10 17:04:03.232174 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:03.232125 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" podStartSLOduration=1.588628961 podStartE2EDuration="4.232111167s" totalImagesPullingTime="2.643482206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:04:03.230295073 +0000 UTC m=+573.117586384" watchObservedRunningTime="2026-07-10 17:04:03.232111167 +0000 UTC m=+573.119402511" Jul 10 17:04:14.202483 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:14.202453 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" Jul 10 17:04:14.202860 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:14.202520 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b46985b8-8jcl7" Jul 10 17:04:14.265624 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:14.265593 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b46985b8-8jcl7"] Jul 10 17:04:14.266037 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:14.266005 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b46985b8-8jcl7" podUID="24f1b9d3-c454-4e09-8707-1505b56970a1" containerName="manager" containerID="cri-o://2b1998c5829cf35db745b8ca9fa80f6f2e71b73e7f65c9a4b6f5e5f593ea0dfa" gracePeriod=30 Jul 10 17:04:14.508015 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:14.507987 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b46985b8-8jcl7" Jul 10 17:04:14.566795 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:14.566764 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24f1b9d3-c454-4e09-8707-1505b56970a1-webhook-cert\") pod \"24f1b9d3-c454-4e09-8707-1505b56970a1\" (UID: \"24f1b9d3-c454-4e09-8707-1505b56970a1\") " Jul 10 17:04:14.566795 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:14.566799 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swngq\" (UniqueName: \"kubernetes.io/projected/24f1b9d3-c454-4e09-8707-1505b56970a1-kube-api-access-swngq\") pod \"24f1b9d3-c454-4e09-8707-1505b56970a1\" (UID: \"24f1b9d3-c454-4e09-8707-1505b56970a1\") " Jul 10 17:04:14.569100 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:14.569062 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f1b9d3-c454-4e09-8707-1505b56970a1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "24f1b9d3-c454-4e09-8707-1505b56970a1" (UID: "24f1b9d3-c454-4e09-8707-1505b56970a1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:04:14.569100 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:14.569088 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f1b9d3-c454-4e09-8707-1505b56970a1-kube-api-access-swngq" (OuterVolumeSpecName: "kube-api-access-swngq") pod "24f1b9d3-c454-4e09-8707-1505b56970a1" (UID: "24f1b9d3-c454-4e09-8707-1505b56970a1"). InnerVolumeSpecName "kube-api-access-swngq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:04:14.667489 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:14.667458 2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24f1b9d3-c454-4e09-8707-1505b56970a1-webhook-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:04:14.667489 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:14.667485 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swngq\" (UniqueName: \"kubernetes.io/projected/24f1b9d3-c454-4e09-8707-1505b56970a1-kube-api-access-swngq\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:04:15.249430 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:15.249391 2567 generic.go:356] "Generic (PLEG): container finished" podID="24f1b9d3-c454-4e09-8707-1505b56970a1" containerID="2b1998c5829cf35db745b8ca9fa80f6f2e71b73e7f65c9a4b6f5e5f593ea0dfa" exitCode=0 Jul 10 17:04:15.249840 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:15.249455 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b46985b8-8jcl7" Jul 10 17:04:15.249840 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:15.249488 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b46985b8-8jcl7" event={"ID":"24f1b9d3-c454-4e09-8707-1505b56970a1","Type":"ContainerDied","Data":"2b1998c5829cf35db745b8ca9fa80f6f2e71b73e7f65c9a4b6f5e5f593ea0dfa"} Jul 10 17:04:15.249840 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:15.249578 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b46985b8-8jcl7" event={"ID":"24f1b9d3-c454-4e09-8707-1505b56970a1","Type":"ContainerDied","Data":"1c5ee770ea227336f9e359ac8473a0776ec647e8995ab0ac26dfad3b8d9fcd64"} Jul 10 17:04:15.249840 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:15.249607 2567 scope.go:117] "RemoveContainer" containerID="2b1998c5829cf35db745b8ca9fa80f6f2e71b73e7f65c9a4b6f5e5f593ea0dfa" Jul 10 17:04:15.258058 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:15.258041 2567 scope.go:117] "RemoveContainer" containerID="2b1998c5829cf35db745b8ca9fa80f6f2e71b73e7f65c9a4b6f5e5f593ea0dfa" Jul 10 17:04:15.258334 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:04:15.258314 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1998c5829cf35db745b8ca9fa80f6f2e71b73e7f65c9a4b6f5e5f593ea0dfa\": container with ID starting with 2b1998c5829cf35db745b8ca9fa80f6f2e71b73e7f65c9a4b6f5e5f593ea0dfa not found: ID does not exist" containerID="2b1998c5829cf35db745b8ca9fa80f6f2e71b73e7f65c9a4b6f5e5f593ea0dfa" Jul 10 17:04:15.258409 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:15.258348 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1998c5829cf35db745b8ca9fa80f6f2e71b73e7f65c9a4b6f5e5f593ea0dfa"} err="failed to get container status \"2b1998c5829cf35db745b8ca9fa80f6f2e71b73e7f65c9a4b6f5e5f593ea0dfa\": rpc error: code = NotFound desc = could not find container \"2b1998c5829cf35db745b8ca9fa80f6f2e71b73e7f65c9a4b6f5e5f593ea0dfa\": container with ID starting with 2b1998c5829cf35db745b8ca9fa80f6f2e71b73e7f65c9a4b6f5e5f593ea0dfa not found: ID does not exist" Jul 10 17:04:15.267004 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:15.266982 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b46985b8-8jcl7"] Jul 10 17:04:15.273033 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:15.273012 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b46985b8-8jcl7"] Jul 10 17:04:16.681833 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:16.681797 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24f1b9d3-c454-4e09-8707-1505b56970a1" path="/var/lib/kubelet/pods/24f1b9d3-c454-4e09-8707-1505b56970a1/volumes" Jul 10 17:04:30.533051 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:30.533019 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:04:30.533578 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:30.533088 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:04:34.713623 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.713580 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7754899c-6glkr"] Jul 10 17:04:34.714085 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.714060 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f1b9d3-c454-4e09-8707-1505b56970a1" containerName="manager" Jul 10 17:04:34.714085 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.714079 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f1b9d3-c454-4e09-8707-1505b56970a1" containerName="manager" Jul 10 17:04:34.714199 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.714177 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="24f1b9d3-c454-4e09-8707-1505b56970a1" containerName="manager" Jul 10 17:04:34.716403 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.716383 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" Jul 10 17:04:34.718771 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.718751 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7zdln\"" Jul 10 17:04:34.718919 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.718901 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 10 17:04:34.729065 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.729045 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7754899c-6glkr"] Jul 10 17:04:34.851246 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.851199 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkzr4\" (UniqueName: \"kubernetes.io/projected/c135b866-8f99-4008-b9fe-3fb5576c1da8-kube-api-access-pkzr4\") pod \"maas-api-7f7754899c-6glkr\" (UID: \"c135b866-8f99-4008-b9fe-3fb5576c1da8\") " pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" Jul 10 17:04:34.851479 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.851389 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c135b866-8f99-4008-b9fe-3fb5576c1da8-maas-api-tls\") pod \"maas-api-7f7754899c-6glkr\" (UID: \"c135b866-8f99-4008-b9fe-3fb5576c1da8\") " pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" Jul 10 17:04:34.851479 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.851417 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c135b866-8f99-4008-b9fe-3fb5576c1da8-openshift-service-ca\") pod \"maas-api-7f7754899c-6glkr\" (UID: \"c135b866-8f99-4008-b9fe-3fb5576c1da8\") " pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" Jul 10 17:04:34.952181 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.952138 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c135b866-8f99-4008-b9fe-3fb5576c1da8-maas-api-tls\") pod \"maas-api-7f7754899c-6glkr\" (UID: \"c135b866-8f99-4008-b9fe-3fb5576c1da8\") " pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" Jul 10 17:04:34.952435 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.952202 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c135b866-8f99-4008-b9fe-3fb5576c1da8-openshift-service-ca\") pod \"maas-api-7f7754899c-6glkr\" (UID: \"c135b866-8f99-4008-b9fe-3fb5576c1da8\") " pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" Jul 10 17:04:34.952435 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.952326 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkzr4\" (UniqueName: \"kubernetes.io/projected/c135b866-8f99-4008-b9fe-3fb5576c1da8-kube-api-access-pkzr4\") pod \"maas-api-7f7754899c-6glkr\" (UID: \"c135b866-8f99-4008-b9fe-3fb5576c1da8\") " pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" Jul 10 17:04:34.953090 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.953063 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c135b866-8f99-4008-b9fe-3fb5576c1da8-openshift-service-ca\") pod \"maas-api-7f7754899c-6glkr\" (UID: \"c135b866-8f99-4008-b9fe-3fb5576c1da8\") " pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" Jul 10 17:04:34.955341 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.955311 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c135b866-8f99-4008-b9fe-3fb5576c1da8-maas-api-tls\") pod \"maas-api-7f7754899c-6glkr\" (UID: \"c135b866-8f99-4008-b9fe-3fb5576c1da8\") " pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" Jul 10 17:04:34.960616 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:34.960594 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkzr4\" (UniqueName: \"kubernetes.io/projected/c135b866-8f99-4008-b9fe-3fb5576c1da8-kube-api-access-pkzr4\") pod \"maas-api-7f7754899c-6glkr\" (UID: \"c135b866-8f99-4008-b9fe-3fb5576c1da8\") " pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" Jul 10 17:04:35.026807 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.026721 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" Jul 10 17:04:35.174192 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.174157 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7754899c-6glkr"] Jul 10 17:04:35.178014 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:04:35.177989 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc135b866_8f99_4008_b9fe_3fb5576c1da8.slice/crio-4abd378c4aa8d55ae1bdad885f1936095f8e22540c0aaf6b438b823b3550683f WatchSource:0}: Error finding container 4abd378c4aa8d55ae1bdad885f1936095f8e22540c0aaf6b438b823b3550683f: Status 404 returned error can't find the container with id 4abd378c4aa8d55ae1bdad885f1936095f8e22540c0aaf6b438b823b3550683f Jul 10 17:04:35.222196 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.222157 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-pdwpf"] Jul 10 17:04:35.225434 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.225412 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-pdwpf" Jul 10 17:04:35.227883 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.227864 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 17:04:35.228000 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.227865 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-x8dwt\"" Jul 10 17:04:35.235379 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.235353 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-pdwpf"] Jul 10 17:04:35.241491 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.241371 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-2c5fz"] Jul 10 17:04:35.244872 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.244850 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-2c5fz" Jul 10 17:04:35.256462 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.256437 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-2c5fz"] Jul 10 17:04:35.348928 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.348833 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" event={"ID":"c135b866-8f99-4008-b9fe-3fb5576c1da8","Type":"ContainerStarted","Data":"4abd378c4aa8d55ae1bdad885f1936095f8e22540c0aaf6b438b823b3550683f"} Jul 10 17:04:35.355404 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.355367 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de535c35-7176-4946-af88-727d337e2269-plugins-config-volume\") pod \"payload-pre-processing-7964457696-pdwpf\" (UID: \"de535c35-7176-4946-af88-727d337e2269\") " pod="openshift-ingress/payload-pre-processing-7964457696-pdwpf" Jul 10 17:04:35.355584 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.355419 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f-plugins-config-volume\") pod \"payload-processing-58f86c7798-2c5fz\" (UID: \"d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f\") " pod="openshift-ingress/payload-processing-58f86c7798-2c5fz" Jul 10 17:04:35.355584 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.355437 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fntp8\" (UniqueName: \"kubernetes.io/projected/d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f-kube-api-access-fntp8\") pod \"payload-processing-58f86c7798-2c5fz\" (UID: \"d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f\") " pod="openshift-ingress/payload-processing-58f86c7798-2c5fz" Jul 10 17:04:35.355584 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.355491 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwbxl\" (UniqueName: \"kubernetes.io/projected/de535c35-7176-4946-af88-727d337e2269-kube-api-access-dwbxl\") pod \"payload-pre-processing-7964457696-pdwpf\" (UID: \"de535c35-7176-4946-af88-727d337e2269\") " pod="openshift-ingress/payload-pre-processing-7964457696-pdwpf" Jul 10 17:04:35.456456 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.456423 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de535c35-7176-4946-af88-727d337e2269-plugins-config-volume\") pod \"payload-pre-processing-7964457696-pdwpf\" (UID: \"de535c35-7176-4946-af88-727d337e2269\") " pod="openshift-ingress/payload-pre-processing-7964457696-pdwpf" Jul 10 17:04:35.456634 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.456475 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f-plugins-config-volume\") pod \"payload-processing-58f86c7798-2c5fz\" (UID: \"d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f\") " pod="openshift-ingress/payload-processing-58f86c7798-2c5fz" Jul 10 17:04:35.456634 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.456494 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fntp8\" (UniqueName: \"kubernetes.io/projected/d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f-kube-api-access-fntp8\") pod \"payload-processing-58f86c7798-2c5fz\" (UID: \"d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f\") " pod="openshift-ingress/payload-processing-58f86c7798-2c5fz" Jul 10 17:04:35.456634 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.456512 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwbxl\" (UniqueName: \"kubernetes.io/projected/de535c35-7176-4946-af88-727d337e2269-kube-api-access-dwbxl\") pod \"payload-pre-processing-7964457696-pdwpf\" (UID: \"de535c35-7176-4946-af88-727d337e2269\") " pod="openshift-ingress/payload-pre-processing-7964457696-pdwpf" Jul 10 17:04:35.457244 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.457213 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f-plugins-config-volume\") pod \"payload-processing-58f86c7798-2c5fz\" (UID: \"d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f\") " pod="openshift-ingress/payload-processing-58f86c7798-2c5fz" Jul 10 17:04:35.457244 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.457218 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de535c35-7176-4946-af88-727d337e2269-plugins-config-volume\") pod \"payload-pre-processing-7964457696-pdwpf\" (UID: \"de535c35-7176-4946-af88-727d337e2269\") " pod="openshift-ingress/payload-pre-processing-7964457696-pdwpf" Jul 10 17:04:35.466308 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.466256 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fntp8\" (UniqueName: \"kubernetes.io/projected/d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f-kube-api-access-fntp8\") pod \"payload-processing-58f86c7798-2c5fz\" (UID: \"d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f\") " pod="openshift-ingress/payload-processing-58f86c7798-2c5fz" Jul 10 17:04:35.467893 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.467867 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwbxl\" (UniqueName: \"kubernetes.io/projected/de535c35-7176-4946-af88-727d337e2269-kube-api-access-dwbxl\") pod \"payload-pre-processing-7964457696-pdwpf\" (UID: \"de535c35-7176-4946-af88-727d337e2269\") " pod="openshift-ingress/payload-pre-processing-7964457696-pdwpf" Jul 10 17:04:35.538097 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.538029 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-pdwpf" Jul 10 17:04:35.554887 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.554853 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-2c5fz" Jul 10 17:04:35.686638 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.686612 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-pdwpf"] Jul 10 17:04:35.689474 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:04:35.689434 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde535c35_7176_4946_af88_727d337e2269.slice/crio-7d90fea407e50534aa6bd5f8dbb811f96259f9cf858e934bf8c813ba3de18751 WatchSource:0}: Error finding container 7d90fea407e50534aa6bd5f8dbb811f96259f9cf858e934bf8c813ba3de18751: Status 404 returned error can't find the container with id 7d90fea407e50534aa6bd5f8dbb811f96259f9cf858e934bf8c813ba3de18751 Jul 10 17:04:35.715137 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:35.715114 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-2c5fz"] Jul 10 17:04:35.716969 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:04:35.716937 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd700cc88_6d5d_4fb5_8c73_c5869b2bbd4f.slice/crio-191ba395ea5f6cd6cabd53389c209f555804387e21f72f32a123712708191096 WatchSource:0}: Error finding container 191ba395ea5f6cd6cabd53389c209f555804387e21f72f32a123712708191096: Status 404 returned error can't find the container with id 191ba395ea5f6cd6cabd53389c209f555804387e21f72f32a123712708191096 Jul 10 17:04:36.355569 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:36.355534 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-pdwpf" event={"ID":"de535c35-7176-4946-af88-727d337e2269","Type":"ContainerStarted","Data":"7d90fea407e50534aa6bd5f8dbb811f96259f9cf858e934bf8c813ba3de18751"} Jul 10 17:04:36.356798 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:36.356763 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-2c5fz" event={"ID":"d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f","Type":"ContainerStarted","Data":"191ba395ea5f6cd6cabd53389c209f555804387e21f72f32a123712708191096"} Jul 10 17:04:38.368169 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:38.368127 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-pdwpf" event={"ID":"de535c35-7176-4946-af88-727d337e2269","Type":"ContainerStarted","Data":"5871571373feb40760c05aa61e86c65c3caf4194e97744d62d46b087a7f3cd64"} Jul 10 17:04:38.368693 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:38.368192 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-pdwpf" Jul 10 17:04:38.369802 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:38.369773 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-2c5fz" event={"ID":"d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f","Type":"ContainerStarted","Data":"48d7cc05f87a0b07d72cb03b75ba98df68815e1846965db2ef4b0d6d4577807a"} Jul 10 17:04:38.369919 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:38.369825 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-2c5fz" Jul 10 17:04:38.371139 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:38.371118 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" event={"ID":"c135b866-8f99-4008-b9fe-3fb5576c1da8","Type":"ContainerStarted","Data":"7dff085d0767a0ce846d1b6394b706a019d5a1b3535444c2477af59a36759543"} Jul 10 17:04:38.371236 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:38.371164 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" Jul 10 17:04:38.409179 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:38.409125 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-pdwpf" podStartSLOduration=1.450198757 podStartE2EDuration="3.409109528s" totalImagesPullingTime="1.958910771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:04:38.392316871 +0000 UTC m=+608.279608195" watchObservedRunningTime="2026-07-10 17:04:38.409109528 +0000 UTC m=+608.296400889" Jul 10 17:04:38.426566 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:38.426510 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" podStartSLOduration=1.958062653 podStartE2EDuration="4.426488758s" totalImagesPullingTime="2.468426105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:04:38.424589672 +0000 UTC m=+608.311880996" watchObservedRunningTime="2026-07-10 17:04:38.426488758 +0000 UTC m=+608.313780081" Jul 10 17:04:38.427701 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:38.427668 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-2c5fz" podStartSLOduration=1.4983097779999999 podStartE2EDuration="3.427653363s" totalImagesPullingTime="1.929343585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:04:38.409390234 +0000 UTC m=+608.296681554" watchObservedRunningTime="2026-07-10 17:04:38.427653363 +0000 UTC m=+608.314944689" Jul 10 17:04:43.401165 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:43.401128 2567 generic.go:356] "Generic (PLEG): container finished" podID="f43b2410-1f24-432b-9437-b7010fe00be2" containerID="566efa65d07bb4786cbc66f0ed36d7d302ed034b6f216ad53a55de20f20e76d8" exitCode=0 Jul 10 17:04:43.401618 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:43.401201 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" event={"ID":"f43b2410-1f24-432b-9437-b7010fe00be2","Type":"ContainerDied","Data":"566efa65d07bb4786cbc66f0ed36d7d302ed034b6f216ad53a55de20f20e76d8"} Jul 10 17:04:43.401618 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:43.401615 2567 scope.go:117] "RemoveContainer" containerID="566efa65d07bb4786cbc66f0ed36d7d302ed034b6f216ad53a55de20f20e76d8" Jul 10 17:04:43.402556 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:43.402530 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:04:43.402623 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:43.402598 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:04:43.402661 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:43.402628 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:04:44.391614 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:44.391569 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f7754899c-6glkr" Jul 10 17:04:44.408012 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:44.407975 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" event={"ID":"f43b2410-1f24-432b-9437-b7010fe00be2","Type":"ContainerStarted","Data":"55cbba3a0584a7b7b188912a37238d405f220f44a9369a19ac034170f634cd26"} Jul 10 17:04:44.629119 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:44.629083 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:04:44.629342 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:44.629258 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-w7rph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.36:15021/healthz/ready\": dial tcp 10.135.0.36:15021: connect: connection refused" start-of-body= Jul 10 17:04:44.629430 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:44.629354 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" podUID="f43b2410-1f24-432b-9437-b7010fe00be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.36:15021/healthz/ready\": dial tcp 10.135.0.36:15021: connect: connection refused" Jul 10 17:04:45.629478 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:45.629435 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-w7rph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.36:15021/healthz/ready\": dial tcp 10.135.0.36:15021: connect: connection refused" start-of-body= Jul 10 17:04:45.629884 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:45.629515 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" podUID="f43b2410-1f24-432b-9437-b7010fe00be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.36:15021/healthz/ready\": dial tcp 10.135.0.36:15021: connect: connection refused" Jul 10 17:04:46.629287 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:46.629236 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-w7rph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.36:15021/healthz/ready\": dial tcp 10.135.0.36:15021: connect: connection refused" start-of-body= Jul 10 17:04:46.629453 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:46.629334 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" podUID="f43b2410-1f24-432b-9437-b7010fe00be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.36:15021/healthz/ready\": dial tcp 10.135.0.36:15021: connect: connection refused" Jul 10 17:04:47.629207 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:47.629172 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-w7rph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.36:15021/healthz/ready\": dial tcp 10.135.0.36:15021: connect: connection refused" start-of-body= Jul 10 17:04:47.629637 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:47.629229 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" podUID="f43b2410-1f24-432b-9437-b7010fe00be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.36:15021/healthz/ready\": dial tcp 10.135.0.36:15021: connect: connection refused" Jul 10 17:04:48.633500 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:48.633470 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:04:48.633871 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:48.633747 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:04:48.634374 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:48.634358 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7rph" Jul 10 17:04:49.380377 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:49.380342 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-2c5fz" Jul 10 17:04:49.380544 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:04:49.380515 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-pdwpf" Jul 10 17:05:01.252000 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.251918 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn"] Jul 10 17:05:01.255299 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.255246 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.257818 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.257778 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 17:05:01.257989 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.257967 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 10 17:05:01.258793 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.258770 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 17:05:01.258923 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.258788 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9gt4g\"" Jul 10 17:05:01.264801 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.264777 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn"] Jul 10 17:05:01.306055 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.306022 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f3a27e0-516b-41eb-98aa-ba02855ffda2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.306055 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.306061 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f3a27e0-516b-41eb-98aa-ba02855ffda2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.306295 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.306083 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f3a27e0-516b-41eb-98aa-ba02855ffda2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.306295 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.306199 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f3a27e0-516b-41eb-98aa-ba02855ffda2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.306295 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.306229 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlhxp\" (UniqueName: \"kubernetes.io/projected/8f3a27e0-516b-41eb-98aa-ba02855ffda2-kube-api-access-qlhxp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.306458 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.306365 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f3a27e0-516b-41eb-98aa-ba02855ffda2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.407231 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.407190 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f3a27e0-516b-41eb-98aa-ba02855ffda2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.407432 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.407253 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f3a27e0-516b-41eb-98aa-ba02855ffda2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.407432 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.407287 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhxp\" (UniqueName: \"kubernetes.io/projected/8f3a27e0-516b-41eb-98aa-ba02855ffda2-kube-api-access-qlhxp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.407432 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.407342 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f3a27e0-516b-41eb-98aa-ba02855ffda2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.407432 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.407389 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f3a27e0-516b-41eb-98aa-ba02855ffda2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.407432 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.407419 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f3a27e0-516b-41eb-98aa-ba02855ffda2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.407672 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.407605 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f3a27e0-516b-41eb-98aa-ba02855ffda2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.407759 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.407738 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f3a27e0-516b-41eb-98aa-ba02855ffda2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.407829 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.407781 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f3a27e0-516b-41eb-98aa-ba02855ffda2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.409655 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.409631 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f3a27e0-516b-41eb-98aa-ba02855ffda2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.410059 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.410038 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f3a27e0-516b-41eb-98aa-ba02855ffda2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.415299 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.415261 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlhxp\" (UniqueName: \"kubernetes.io/projected/8f3a27e0-516b-41eb-98aa-ba02855ffda2-kube-api-access-qlhxp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn\" (UID: \"8f3a27e0-516b-41eb-98aa-ba02855ffda2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.568710 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.568620 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:01.705303 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.705255 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn"] Jul 10 17:05:01.707895 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:05:01.707863 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3a27e0_516b_41eb_98aa_ba02855ffda2.slice/crio-9332ce12e5a5851beee93be9c2b8a03e2986e6c50b17c9fcfe320ab547e7c438 WatchSource:0}: Error finding container 9332ce12e5a5851beee93be9c2b8a03e2986e6c50b17c9fcfe320ab547e7c438: Status 404 returned error can't find the container with id 9332ce12e5a5851beee93be9c2b8a03e2986e6c50b17c9fcfe320ab547e7c438 Jul 10 17:05:01.709352 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:01.709333 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:05:02.499344 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:02.499307 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" event={"ID":"8f3a27e0-516b-41eb-98aa-ba02855ffda2","Type":"ContainerStarted","Data":"9332ce12e5a5851beee93be9c2b8a03e2986e6c50b17c9fcfe320ab547e7c438"} Jul 10 17:05:03.505643 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:03.505599 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" event={"ID":"8f3a27e0-516b-41eb-98aa-ba02855ffda2","Type":"ContainerStarted","Data":"4b8e5b5453f9f0a42058c436946c34f674b318a0a412d34345484a77d36db3cf"} Jul 10 17:05:03.553291 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:03.553228 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" podStartSLOduration=1.2509998709999999 podStartE2EDuration="2.553210816s" totalImagesPullingTime="1.302210945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:05:03.551914704 +0000 UTC m=+633.439206038" watchObservedRunningTime="2026-07-10 17:05:03.553210816 +0000 UTC m=+633.440502136" Jul 10 17:05:08.749820 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.749779 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst"] Jul 10 17:05:08.753611 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.753583 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.756014 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.755992 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 17:05:08.764651 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.764626 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst"] Jul 10 17:05:08.880575 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.880538 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.880575 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.880578 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.880829 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.880616 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmnsn\" (UniqueName: \"kubernetes.io/projected/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-kube-api-access-cmnsn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.880829 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.880703 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.880829 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.880751 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.880935 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.880844 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.981801 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.981756 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.981801 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.981805 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.982126 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.981844 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmnsn\" (UniqueName: \"kubernetes.io/projected/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-kube-api-access-cmnsn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.982126 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.981871 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.982126 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.981901 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.982126 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.981970 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.982339 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.982265 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.982339 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.982301 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.982440 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.982400 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.984475 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.984455 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.984562 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.984545 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:08.990492 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:08.990456 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmnsn\" (UniqueName: \"kubernetes.io/projected/ecafcbdd-d66f-4652-aead-ed9b9cd07a62-kube-api-access-cmnsn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6fgst\" (UID: \"ecafcbdd-d66f-4652-aead-ed9b9cd07a62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:09.064889 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:09.064802 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:09.206780 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:09.206752 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst"] Jul 10 17:05:09.209393 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:05:09.209358 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecafcbdd_d66f_4652_aead_ed9b9cd07a62.slice/crio-2c310ca0c4f12db4412160536accd8580efc9b2a0fea67d97b9f3951d4213967 WatchSource:0}: Error finding container 2c310ca0c4f12db4412160536accd8580efc9b2a0fea67d97b9f3951d4213967: Status 404 returned error can't find the container with id 2c310ca0c4f12db4412160536accd8580efc9b2a0fea67d97b9f3951d4213967 Jul 10 17:05:09.535688 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:09.535654 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" event={"ID":"ecafcbdd-d66f-4652-aead-ed9b9cd07a62","Type":"ContainerStarted","Data":"acda076e1ddfb88b7bf8de9cbb8e386b053f64e5f343adf2f5bdfba59b350e92"} Jul 10 17:05:09.535858 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:09.535708 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" event={"ID":"ecafcbdd-d66f-4652-aead-ed9b9cd07a62","Type":"ContainerStarted","Data":"2c310ca0c4f12db4412160536accd8580efc9b2a0fea67d97b9f3951d4213967"} Jul 10 17:05:09.552832 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:09.552766 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" podStartSLOduration=1.299467068 podStartE2EDuration="1.552748774s" totalImagesPullingTime="253.281706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:05:09.551438574 +0000 UTC m=+639.438729895" watchObservedRunningTime="2026-07-10 17:05:09.552748774 +0000 UTC m=+639.440040094" Jul 10 17:05:11.568899 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:11.568864 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:11.569433 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:11.568927 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:11.585163 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:11.585128 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:12.561698 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:12.561661 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn" Jul 10 17:05:19.065504 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:19.065441 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:19.065966 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:19.065530 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:19.078677 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:19.078647 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:19.594899 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:19.594861 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6fgst" Jul 10 17:05:30.237870 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.237791 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw"] Jul 10 17:05:30.241319 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.241292 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.243768 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.243745 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 17:05:30.249422 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.249394 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw"] Jul 10 17:05:30.285824 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.285790 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1685e004-b215-4793-b784-c477c2df1823-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.285983 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.285836 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1685e004-b215-4793-b784-c477c2df1823-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.285983 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.285889 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1685e004-b215-4793-b784-c477c2df1823-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.285983 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.285926 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f448k\" (UniqueName: \"kubernetes.io/projected/1685e004-b215-4793-b784-c477c2df1823-kube-api-access-f448k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.285983 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.285971 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1685e004-b215-4793-b784-c477c2df1823-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.286138 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.285991 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1685e004-b215-4793-b784-c477c2df1823-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.386463 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.386422 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1685e004-b215-4793-b784-c477c2df1823-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.386463 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.386469 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1685e004-b215-4793-b784-c477c2df1823-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.386699 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.386487 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1685e004-b215-4793-b784-c477c2df1823-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.386699 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.386602 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f448k\" (UniqueName: \"kubernetes.io/projected/1685e004-b215-4793-b784-c477c2df1823-kube-api-access-f448k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.386699 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.386653 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1685e004-b215-4793-b784-c477c2df1823-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.386699 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.386682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1685e004-b215-4793-b784-c477c2df1823-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.387079 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.387013 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1685e004-b215-4793-b784-c477c2df1823-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.387079 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.387070 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1685e004-b215-4793-b784-c477c2df1823-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.387287 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.387108 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1685e004-b215-4793-b784-c477c2df1823-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.388967 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.388940 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1685e004-b215-4793-b784-c477c2df1823-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.389248 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.389227 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1685e004-b215-4793-b784-c477c2df1823-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.394123 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.394098 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f448k\" (UniqueName: \"kubernetes.io/projected/1685e004-b215-4793-b784-c477c2df1823-kube-api-access-f448k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw\" (UID: \"1685e004-b215-4793-b784-c477c2df1823\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.554150 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.554070 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:30.700264 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:30.700239 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw"] Jul 10 17:05:30.702518 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:05:30.702487 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1685e004_b215_4793_b784_c477c2df1823.slice/crio-69110b40273ddfd237594e095a5f791a2e7ce98b96203c6f87842b14f9a0cb5a WatchSource:0}: Error finding container 69110b40273ddfd237594e095a5f791a2e7ce98b96203c6f87842b14f9a0cb5a: Status 404 returned error can't find the container with id 69110b40273ddfd237594e095a5f791a2e7ce98b96203c6f87842b14f9a0cb5a Jul 10 17:05:31.648861 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:31.648822 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" event={"ID":"1685e004-b215-4793-b784-c477c2df1823","Type":"ContainerStarted","Data":"69941607796d18198138688df5ba426bc87dabd90206fb8bee81edd4e7f13b59"} Jul 10 17:05:31.649305 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:31.648885 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" event={"ID":"1685e004-b215-4793-b784-c477c2df1823","Type":"ContainerStarted","Data":"69110b40273ddfd237594e095a5f791a2e7ce98b96203c6f87842b14f9a0cb5a"} Jul 10 17:05:31.668775 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:31.668715 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" podStartSLOduration=1.4223187130000001 podStartE2EDuration="1.668679298s" totalImagesPullingTime="246.360585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:05:31.666986687 +0000 UTC m=+661.554278006" watchObservedRunningTime="2026-07-10 17:05:31.668679298 +0000 UTC m=+661.555970619" Jul 10 17:05:40.554514 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:40.554465 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:40.555000 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:40.554544 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:40.567233 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:40.567203 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:05:40.702934 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:05:40.702902 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw" Jul 10 17:06:10.848524 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.848492 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp"] Jul 10 17:06:10.852607 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.852587 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.855208 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.855185 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 10 17:06:10.862480 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.862448 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp"] Jul 10 17:06:10.873971 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.873942 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/747ba32a-9f68-4893-843a-5f2798a893f1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.874099 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.873981 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/747ba32a-9f68-4893-843a-5f2798a893f1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.874141 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.874097 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/747ba32a-9f68-4893-843a-5f2798a893f1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.874175 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.874133 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/747ba32a-9f68-4893-843a-5f2798a893f1-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.874212 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.874191 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxq5c\" (UniqueName: \"kubernetes.io/projected/747ba32a-9f68-4893-843a-5f2798a893f1-kube-api-access-bxq5c\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.874246 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.874223 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/747ba32a-9f68-4893-843a-5f2798a893f1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.974783 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.974745 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/747ba32a-9f68-4893-843a-5f2798a893f1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.974991 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.974793 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/747ba32a-9f68-4893-843a-5f2798a893f1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.974991 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.974889 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/747ba32a-9f68-4893-843a-5f2798a893f1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.974991 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.974924 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/747ba32a-9f68-4893-843a-5f2798a893f1-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.974991 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.974981 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxq5c\" (UniqueName: \"kubernetes.io/projected/747ba32a-9f68-4893-843a-5f2798a893f1-kube-api-access-bxq5c\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.975513 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.975015 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/747ba32a-9f68-4893-843a-5f2798a893f1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.975513 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.975386 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/747ba32a-9f68-4893-843a-5f2798a893f1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.975513 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.975413 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/747ba32a-9f68-4893-843a-5f2798a893f1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.975663 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.975557 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/747ba32a-9f68-4893-843a-5f2798a893f1-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.977560 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.977534 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/747ba32a-9f68-4893-843a-5f2798a893f1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.977757 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.977732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/747ba32a-9f68-4893-843a-5f2798a893f1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:10.989246 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:10.989218 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxq5c\" (UniqueName: \"kubernetes.io/projected/747ba32a-9f68-4893-843a-5f2798a893f1-kube-api-access-bxq5c\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp\" (UID: \"747ba32a-9f68-4893-843a-5f2798a893f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:11.163937 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:11.163847 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:11.303336 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:11.303302 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp"] Jul 10 17:06:11.305005 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:06:11.304979 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod747ba32a_9f68_4893_843a_5f2798a893f1.slice/crio-a1fc4c3d1820dd5fcab5dbae06e67380f00ac53d7ab89d76f55334936fbc9b70 WatchSource:0}: Error finding container a1fc4c3d1820dd5fcab5dbae06e67380f00ac53d7ab89d76f55334936fbc9b70: Status 404 returned error can't find the container with id a1fc4c3d1820dd5fcab5dbae06e67380f00ac53d7ab89d76f55334936fbc9b70 Jul 10 17:06:11.851095 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:11.851004 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" event={"ID":"747ba32a-9f68-4893-843a-5f2798a893f1","Type":"ContainerStarted","Data":"c764ce51c54d1e158578f489b0c5117a2add2a7e947965f98e3a59e673c56e4f"} Jul 10 17:06:11.851095 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:11.851051 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" event={"ID":"747ba32a-9f68-4893-843a-5f2798a893f1","Type":"ContainerStarted","Data":"a1fc4c3d1820dd5fcab5dbae06e67380f00ac53d7ab89d76f55334936fbc9b70"} Jul 10 17:06:11.868233 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:11.868184 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" podStartSLOduration=1.6081477020000001 podStartE2EDuration="1.868168649s" totalImagesPullingTime="260.020947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:06:11.866571494 +0000 UTC m=+701.753862814" watchObservedRunningTime="2026-07-10 17:06:11.868168649 +0000 UTC m=+701.755459968" Jul 10 17:06:21.164995 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:21.164906 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:21.164995 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:21.164955 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:21.177760 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:21.177737 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:21.912489 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:21.912457 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp" Jul 10 17:06:55.930213 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:55.930173 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c4668c64b-7tjvg"] Jul 10 17:06:55.933809 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:55.933789 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4668c64b-7tjvg" Jul 10 17:06:55.941390 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:55.941368 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c4668c64b-7tjvg"] Jul 10 17:06:55.981186 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:55.981157 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7107d3ea-1665-470b-8424-c8c9734d1ae3-tls-cert\") pod \"authorino-5c4668c64b-7tjvg\" (UID: \"7107d3ea-1665-470b-8424-c8c9734d1ae3\") " pod="kuadrant-system/authorino-5c4668c64b-7tjvg" Jul 10 17:06:55.981348 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:55.981236 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8c9\" (UniqueName: \"kubernetes.io/projected/7107d3ea-1665-470b-8424-c8c9734d1ae3-kube-api-access-lj8c9\") pod \"authorino-5c4668c64b-7tjvg\" (UID: \"7107d3ea-1665-470b-8424-c8c9734d1ae3\") " pod="kuadrant-system/authorino-5c4668c64b-7tjvg" Jul 10 17:06:56.082238 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:56.082201 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8c9\" (UniqueName: \"kubernetes.io/projected/7107d3ea-1665-470b-8424-c8c9734d1ae3-kube-api-access-lj8c9\") pod \"authorino-5c4668c64b-7tjvg\" (UID: \"7107d3ea-1665-470b-8424-c8c9734d1ae3\") " pod="kuadrant-system/authorino-5c4668c64b-7tjvg" Jul 10 17:06:56.082448 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:56.082314 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7107d3ea-1665-470b-8424-c8c9734d1ae3-tls-cert\") pod \"authorino-5c4668c64b-7tjvg\" (UID: \"7107d3ea-1665-470b-8424-c8c9734d1ae3\") " pod="kuadrant-system/authorino-5c4668c64b-7tjvg" Jul 10 17:06:56.085035 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:56.085008 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7107d3ea-1665-470b-8424-c8c9734d1ae3-tls-cert\") pod \"authorino-5c4668c64b-7tjvg\" (UID: \"7107d3ea-1665-470b-8424-c8c9734d1ae3\") " pod="kuadrant-system/authorino-5c4668c64b-7tjvg" Jul 10 17:06:56.090146 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:56.090119 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8c9\" (UniqueName: \"kubernetes.io/projected/7107d3ea-1665-470b-8424-c8c9734d1ae3-kube-api-access-lj8c9\") pod \"authorino-5c4668c64b-7tjvg\" (UID: \"7107d3ea-1665-470b-8424-c8c9734d1ae3\") " pod="kuadrant-system/authorino-5c4668c64b-7tjvg" Jul 10 17:06:56.244362 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:56.244327 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4668c64b-7tjvg" Jul 10 17:06:56.378397 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:56.378369 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c4668c64b-7tjvg"] Jul 10 17:06:56.382494 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:06:56.382399 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7107d3ea_1665_470b_8424_c8c9734d1ae3.slice/crio-b4bcb5a55d71e2a245d309db13cae73ed6b67170bef6bb89d5ce77364935be57 WatchSource:0}: Error finding container b4bcb5a55d71e2a245d309db13cae73ed6b67170bef6bb89d5ce77364935be57: Status 404 returned error can't find the container with id b4bcb5a55d71e2a245d309db13cae73ed6b67170bef6bb89d5ce77364935be57 Jul 10 17:06:57.072255 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:57.072217 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c4668c64b-7tjvg" event={"ID":"7107d3ea-1665-470b-8424-c8c9734d1ae3","Type":"ContainerStarted","Data":"b4bcb5a55d71e2a245d309db13cae73ed6b67170bef6bb89d5ce77364935be57"} Jul 10 17:06:58.083227 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:58.083186 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c4668c64b-7tjvg" event={"ID":"7107d3ea-1665-470b-8424-c8c9734d1ae3","Type":"ContainerStarted","Data":"ab8b0cfd1a0a5956051a2dbf27c41cce617b6c2e30b43609898956668cefb267"} Jul 10 17:06:58.101613 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:58.101552 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5c4668c64b-7tjvg" podStartSLOduration=2.380769585 podStartE2EDuration="3.101530927s" totalImagesPullingTime="720.761342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:06:58.101134997 +0000 UTC m=+747.988426318" watchObservedRunningTime="2026-07-10 17:06:58.101530927 +0000 UTC m=+747.988822248" Jul 10 17:06:58.131296 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:58.131098 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-564b958767-prqt8"] Jul 10 17:06:58.131476 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:58.131339 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-564b958767-prqt8" podUID="233ed9ef-cdae-4c66-88cc-049bd062935a" containerName="authorino" containerID="cri-o://8a77beeca10d61df056ec289484c086d6ea7f1b2ebc28a17eae23278f12964cb" gracePeriod=30 Jul 10 17:06:58.393850 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:58.393817 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-564b958767-prqt8" Jul 10 17:06:58.403266 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:58.403242 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/233ed9ef-cdae-4c66-88cc-049bd062935a-tls-cert\") pod \"233ed9ef-cdae-4c66-88cc-049bd062935a\" (UID: \"233ed9ef-cdae-4c66-88cc-049bd062935a\") " Jul 10 17:06:58.403402 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:58.403314 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djp4j\" (UniqueName: \"kubernetes.io/projected/233ed9ef-cdae-4c66-88cc-049bd062935a-kube-api-access-djp4j\") pod \"233ed9ef-cdae-4c66-88cc-049bd062935a\" (UID: \"233ed9ef-cdae-4c66-88cc-049bd062935a\") " Jul 10 17:06:58.405647 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:58.405614 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233ed9ef-cdae-4c66-88cc-049bd062935a-kube-api-access-djp4j" (OuterVolumeSpecName: "kube-api-access-djp4j") pod "233ed9ef-cdae-4c66-88cc-049bd062935a" (UID: "233ed9ef-cdae-4c66-88cc-049bd062935a"). InnerVolumeSpecName "kube-api-access-djp4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:06:58.415749 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:58.415725 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233ed9ef-cdae-4c66-88cc-049bd062935a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "233ed9ef-cdae-4c66-88cc-049bd062935a" (UID: "233ed9ef-cdae-4c66-88cc-049bd062935a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:06:58.510189 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:58.504886 2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/233ed9ef-cdae-4c66-88cc-049bd062935a-tls-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:06:58.510189 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:58.504923 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djp4j\" (UniqueName: \"kubernetes.io/projected/233ed9ef-cdae-4c66-88cc-049bd062935a-kube-api-access-djp4j\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:06:59.090091 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:59.089996 2567 generic.go:356] "Generic (PLEG): container finished" podID="233ed9ef-cdae-4c66-88cc-049bd062935a" containerID="8a77beeca10d61df056ec289484c086d6ea7f1b2ebc28a17eae23278f12964cb" exitCode=0 Jul 10 17:06:59.090091 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:59.090067 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-564b958767-prqt8" Jul 10 17:06:59.090611 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:59.090051 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-564b958767-prqt8" event={"ID":"233ed9ef-cdae-4c66-88cc-049bd062935a","Type":"ContainerDied","Data":"8a77beeca10d61df056ec289484c086d6ea7f1b2ebc28a17eae23278f12964cb"} Jul 10 17:06:59.090611 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:59.090172 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-564b958767-prqt8" event={"ID":"233ed9ef-cdae-4c66-88cc-049bd062935a","Type":"ContainerDied","Data":"57f178ffe34353c58c83975fa1dc7f2af1952bf526bf047541748e605675331b"} Jul 10 17:06:59.090611 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:59.090217 2567 scope.go:117] "RemoveContainer" containerID="8a77beeca10d61df056ec289484c086d6ea7f1b2ebc28a17eae23278f12964cb" Jul 10 17:06:59.100812 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:59.100792 2567 scope.go:117] "RemoveContainer" containerID="8a77beeca10d61df056ec289484c086d6ea7f1b2ebc28a17eae23278f12964cb" Jul 10 17:06:59.101134 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:06:59.101108 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a77beeca10d61df056ec289484c086d6ea7f1b2ebc28a17eae23278f12964cb\": container with ID starting with 8a77beeca10d61df056ec289484c086d6ea7f1b2ebc28a17eae23278f12964cb not found: ID does not exist" containerID="8a77beeca10d61df056ec289484c086d6ea7f1b2ebc28a17eae23278f12964cb" Jul 10 17:06:59.101203 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:59.101143 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a77beeca10d61df056ec289484c086d6ea7f1b2ebc28a17eae23278f12964cb"} err="failed to get container status \"8a77beeca10d61df056ec289484c086d6ea7f1b2ebc28a17eae23278f12964cb\": rpc error: code = NotFound desc = could not find container \"8a77beeca10d61df056ec289484c086d6ea7f1b2ebc28a17eae23278f12964cb\": container with ID starting with 8a77beeca10d61df056ec289484c086d6ea7f1b2ebc28a17eae23278f12964cb not found: ID does not exist" Jul 10 17:06:59.108849 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:59.108817 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-564b958767-prqt8"] Jul 10 17:06:59.112663 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:06:59.112635 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-564b958767-prqt8"] Jul 10 17:07:00.683869 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:07:00.683838 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="233ed9ef-cdae-4c66-88cc-049bd062935a" path="/var/lib/kubelet/pods/233ed9ef-cdae-4c66-88cc-049bd062935a/volumes" Jul 10 17:08:10.692120 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:10.692085 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c76c8d8b8-fvbbn"] Jul 10 17:08:10.692647 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:10.692313 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" podUID="b4d91000-362c-4c24-a6a2-5d44a7e91ea1" containerName="manager" containerID="cri-o://fde3f5f60880e57365435ed81621a7bf2389d98b684ead4bfde55f917fe6d333" gracePeriod=30 Jul 10 17:08:10.945738 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:10.945676 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" Jul 10 17:08:11.070258 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.070217 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4d91000-362c-4c24-a6a2-5d44a7e91ea1-webhook-cert\") pod \"b4d91000-362c-4c24-a6a2-5d44a7e91ea1\" (UID: \"b4d91000-362c-4c24-a6a2-5d44a7e91ea1\") " Jul 10 17:08:11.070450 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.070387 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfndw\" (UniqueName: \"kubernetes.io/projected/b4d91000-362c-4c24-a6a2-5d44a7e91ea1-kube-api-access-nfndw\") pod \"b4d91000-362c-4c24-a6a2-5d44a7e91ea1\" (UID: \"b4d91000-362c-4c24-a6a2-5d44a7e91ea1\") " Jul 10 17:08:11.072479 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.072447 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d91000-362c-4c24-a6a2-5d44a7e91ea1-kube-api-access-nfndw" (OuterVolumeSpecName: "kube-api-access-nfndw") pod "b4d91000-362c-4c24-a6a2-5d44a7e91ea1" (UID: "b4d91000-362c-4c24-a6a2-5d44a7e91ea1"). InnerVolumeSpecName "kube-api-access-nfndw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:08:11.072627 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.072553 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d91000-362c-4c24-a6a2-5d44a7e91ea1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b4d91000-362c-4c24-a6a2-5d44a7e91ea1" (UID: "b4d91000-362c-4c24-a6a2-5d44a7e91ea1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:08:11.172066 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.172031 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfndw\" (UniqueName: \"kubernetes.io/projected/b4d91000-362c-4c24-a6a2-5d44a7e91ea1-kube-api-access-nfndw\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:08:11.172066 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.172062 2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4d91000-362c-4c24-a6a2-5d44a7e91ea1-webhook-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:08:11.450664 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.450628 2567 generic.go:356] "Generic (PLEG): container finished" podID="b4d91000-362c-4c24-a6a2-5d44a7e91ea1" containerID="fde3f5f60880e57365435ed81621a7bf2389d98b684ead4bfde55f917fe6d333" exitCode=0 Jul 10 17:08:11.450853 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.450676 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" Jul 10 17:08:11.450853 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.450691 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" event={"ID":"b4d91000-362c-4c24-a6a2-5d44a7e91ea1","Type":"ContainerDied","Data":"fde3f5f60880e57365435ed81621a7bf2389d98b684ead4bfde55f917fe6d333"} Jul 10 17:08:11.450853 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.450735 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c76c8d8b8-fvbbn" event={"ID":"b4d91000-362c-4c24-a6a2-5d44a7e91ea1","Type":"ContainerDied","Data":"682a9f5b41fe39da9b32a741b9f06396cfc109a04627eb5fe475e216b3185a92"} Jul 10 17:08:11.450853 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.450755 2567 scope.go:117] "RemoveContainer" containerID="fde3f5f60880e57365435ed81621a7bf2389d98b684ead4bfde55f917fe6d333" Jul 10 17:08:11.460153 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.460131 2567 scope.go:117] "RemoveContainer" containerID="fde3f5f60880e57365435ed81621a7bf2389d98b684ead4bfde55f917fe6d333" Jul 10 17:08:11.460483 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:08:11.460464 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde3f5f60880e57365435ed81621a7bf2389d98b684ead4bfde55f917fe6d333\": container with ID starting with fde3f5f60880e57365435ed81621a7bf2389d98b684ead4bfde55f917fe6d333 not found: ID does not exist" containerID="fde3f5f60880e57365435ed81621a7bf2389d98b684ead4bfde55f917fe6d333" Jul 10 17:08:11.460569 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.460496 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde3f5f60880e57365435ed81621a7bf2389d98b684ead4bfde55f917fe6d333"} err="failed to get container status \"fde3f5f60880e57365435ed81621a7bf2389d98b684ead4bfde55f917fe6d333\": rpc error: code = NotFound desc = could not find container \"fde3f5f60880e57365435ed81621a7bf2389d98b684ead4bfde55f917fe6d333\": container with ID starting with fde3f5f60880e57365435ed81621a7bf2389d98b684ead4bfde55f917fe6d333 not found: ID does not exist" Jul 10 17:08:11.471465 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.471435 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c76c8d8b8-fvbbn"] Jul 10 17:08:11.474764 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:11.474738 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c76c8d8b8-fvbbn"] Jul 10 17:08:12.286345 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.286305 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c76c8d8b8-b2464"] Jul 10 17:08:12.286895 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.286873 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4d91000-362c-4c24-a6a2-5d44a7e91ea1" containerName="manager" Jul 10 17:08:12.286983 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.286899 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4d91000-362c-4c24-a6a2-5d44a7e91ea1" containerName="manager" Jul 10 17:08:12.286983 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.286924 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233ed9ef-cdae-4c66-88cc-049bd062935a" containerName="authorino" Jul 10 17:08:12.286983 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.286932 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="233ed9ef-cdae-4c66-88cc-049bd062935a" containerName="authorino" Jul 10 17:08:12.287133 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.287031 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="233ed9ef-cdae-4c66-88cc-049bd062935a" containerName="authorino" Jul 10 17:08:12.287133 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.287049 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4d91000-362c-4c24-a6a2-5d44a7e91ea1" containerName="manager" Jul 10 17:08:12.291550 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.291526 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c76c8d8b8-b2464" Jul 10 17:08:12.293882 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.293858 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 17:08:12.294607 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.294581 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dr5rg\"" Jul 10 17:08:12.294738 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.294654 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 17:08:12.297112 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.297090 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c76c8d8b8-b2464"] Jul 10 17:08:12.383964 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.383918 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a2202f3-4809-4b88-8837-0614397cb2dc-webhook-cert\") pod \"maas-controller-6c76c8d8b8-b2464\" (UID: \"3a2202f3-4809-4b88-8837-0614397cb2dc\") " pod="opendatahub/maas-controller-6c76c8d8b8-b2464" Jul 10 17:08:12.383964 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.383962 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvhbl\" (UniqueName: \"kubernetes.io/projected/3a2202f3-4809-4b88-8837-0614397cb2dc-kube-api-access-fvhbl\") pod \"maas-controller-6c76c8d8b8-b2464\" (UID: \"3a2202f3-4809-4b88-8837-0614397cb2dc\") " pod="opendatahub/maas-controller-6c76c8d8b8-b2464" Jul 10 17:08:12.485507 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.485473 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a2202f3-4809-4b88-8837-0614397cb2dc-webhook-cert\") pod \"maas-controller-6c76c8d8b8-b2464\" (UID: \"3a2202f3-4809-4b88-8837-0614397cb2dc\") " pod="opendatahub/maas-controller-6c76c8d8b8-b2464" Jul 10 17:08:12.485712 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.485513 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvhbl\" (UniqueName: \"kubernetes.io/projected/3a2202f3-4809-4b88-8837-0614397cb2dc-kube-api-access-fvhbl\") pod \"maas-controller-6c76c8d8b8-b2464\" (UID: \"3a2202f3-4809-4b88-8837-0614397cb2dc\") " pod="opendatahub/maas-controller-6c76c8d8b8-b2464" Jul 10 17:08:12.488014 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.487989 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a2202f3-4809-4b88-8837-0614397cb2dc-webhook-cert\") pod \"maas-controller-6c76c8d8b8-b2464\" (UID: \"3a2202f3-4809-4b88-8837-0614397cb2dc\") " pod="opendatahub/maas-controller-6c76c8d8b8-b2464" Jul 10 17:08:12.493065 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.493038 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvhbl\" (UniqueName: \"kubernetes.io/projected/3a2202f3-4809-4b88-8837-0614397cb2dc-kube-api-access-fvhbl\") pod \"maas-controller-6c76c8d8b8-b2464\" (UID: \"3a2202f3-4809-4b88-8837-0614397cb2dc\") " pod="opendatahub/maas-controller-6c76c8d8b8-b2464" Jul 10 17:08:12.603348 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.603255 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c76c8d8b8-b2464" Jul 10 17:08:12.682261 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.682230 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4d91000-362c-4c24-a6a2-5d44a7e91ea1" path="/var/lib/kubelet/pods/b4d91000-362c-4c24-a6a2-5d44a7e91ea1/volumes" Jul 10 17:08:12.741220 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:12.741191 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c76c8d8b8-b2464"] Jul 10 17:08:12.743368 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:08:12.743340 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2202f3_4809_4b88_8837_0614397cb2dc.slice/crio-e1efef2fbbbcf8e4da0ec0df848d8134b627596fc03275dd5bd5948562edfe50 WatchSource:0}: Error finding container e1efef2fbbbcf8e4da0ec0df848d8134b627596fc03275dd5bd5948562edfe50: Status 404 returned error can't find the container with id e1efef2fbbbcf8e4da0ec0df848d8134b627596fc03275dd5bd5948562edfe50 Jul 10 17:08:13.463357 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:13.463326 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c76c8d8b8-b2464" event={"ID":"3a2202f3-4809-4b88-8837-0614397cb2dc","Type":"ContainerStarted","Data":"f9f00b0440d663afdfc25bcaebda81ae4a9493965ab53b500abb1df132e5b137"} Jul 10 17:08:13.463749 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:13.463376 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c76c8d8b8-b2464" event={"ID":"3a2202f3-4809-4b88-8837-0614397cb2dc","Type":"ContainerStarted","Data":"e1efef2fbbbcf8e4da0ec0df848d8134b627596fc03275dd5bd5948562edfe50"} Jul 10 17:08:13.463749 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:13.463411 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c76c8d8b8-b2464" Jul 10 17:08:13.479290 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:13.479209 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c76c8d8b8-b2464" podStartSLOduration=0.959077137 podStartE2EDuration="1.479188481s" totalImagesPullingTime="520.111344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:08:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:08:13.478214813 +0000 UTC m=+823.365506133" watchObservedRunningTime="2026-07-10 17:08:13.479188481 +0000 UTC m=+823.366479802" Jul 10 17:08:24.475017 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:08:24.474984 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c76c8d8b8-b2464" Jul 10 17:09:30.561725 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:09:30.561642 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:09:30.562686 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:09:30.562659 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:13:04.314019 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:04.313982 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv"] Jul 10 17:13:04.316719 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:04.314256 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" podUID="a288f196-9b79-44fa-a961-97c54a6c179d" containerName="manager" containerID="cri-o://044cfb9b0c83d64c2f7f2a33a387771470c819744a800550054c7cf087d1788b" gracePeriod=10 Jul 10 17:13:07.964634 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:07.964596 2567 generic.go:356] "Generic (PLEG): container finished" podID="a288f196-9b79-44fa-a961-97c54a6c179d" containerID="044cfb9b0c83d64c2f7f2a33a387771470c819744a800550054c7cf087d1788b" exitCode=0 Jul 10 17:13:07.965002 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:07.964676 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" event={"ID":"a288f196-9b79-44fa-a961-97c54a6c179d","Type":"ContainerDied","Data":"044cfb9b0c83d64c2f7f2a33a387771470c819744a800550054c7cf087d1788b"} Jul 10 17:13:08.262565 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:08.262542 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" Jul 10 17:13:08.337020 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:08.336991 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a288f196-9b79-44fa-a961-97c54a6c179d-extensions-socket-volume\") pod \"a288f196-9b79-44fa-a961-97c54a6c179d\" (UID: \"a288f196-9b79-44fa-a961-97c54a6c179d\") " Jul 10 17:13:08.337157 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:08.337096 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ngdc\" (UniqueName: \"kubernetes.io/projected/a288f196-9b79-44fa-a961-97c54a6c179d-kube-api-access-8ngdc\") pod \"a288f196-9b79-44fa-a961-97c54a6c179d\" (UID: \"a288f196-9b79-44fa-a961-97c54a6c179d\") " Jul 10 17:13:08.337416 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:08.337393 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a288f196-9b79-44fa-a961-97c54a6c179d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a288f196-9b79-44fa-a961-97c54a6c179d" (UID: "a288f196-9b79-44fa-a961-97c54a6c179d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:13:08.339231 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:08.339202 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a288f196-9b79-44fa-a961-97c54a6c179d-kube-api-access-8ngdc" (OuterVolumeSpecName: "kube-api-access-8ngdc") pod "a288f196-9b79-44fa-a961-97c54a6c179d" (UID: "a288f196-9b79-44fa-a961-97c54a6c179d"). InnerVolumeSpecName "kube-api-access-8ngdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:13:08.438034 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:08.437999 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ngdc\" (UniqueName: \"kubernetes.io/projected/a288f196-9b79-44fa-a961-97c54a6c179d-kube-api-access-8ngdc\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:13:08.438034 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:08.438027 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a288f196-9b79-44fa-a961-97c54a6c179d-extensions-socket-volume\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:13:08.970863 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:08.970829 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" event={"ID":"a288f196-9b79-44fa-a961-97c54a6c179d","Type":"ContainerDied","Data":"5e1c509801236de5e74d75d15b847d8b8993c206bf92b47302d62864f872224b"} Jul 10 17:13:08.970863 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:08.970861 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv" Jul 10 17:13:08.971411 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:08.970888 2567 scope.go:117] "RemoveContainer" containerID="044cfb9b0c83d64c2f7f2a33a387771470c819744a800550054c7cf087d1788b" Jul 10 17:13:08.987794 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:08.987770 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv"] Jul 10 17:13:08.991717 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:08.991694 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rq7zv"] Jul 10 17:13:10.681907 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:13:10.681876 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a288f196-9b79-44fa-a961-97c54a6c179d" path="/var/lib/kubelet/pods/a288f196-9b79-44fa-a961-97c54a6c179d/volumes" Jul 10 17:14:05.694694 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:05.694585 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p"] Jul 10 17:14:05.695324 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:05.695009 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a288f196-9b79-44fa-a961-97c54a6c179d" containerName="manager" Jul 10 17:14:05.695324 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:05.695021 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a288f196-9b79-44fa-a961-97c54a6c179d" containerName="manager" Jul 10 17:14:05.695324 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:05.695099 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a288f196-9b79-44fa-a961-97c54a6c179d" containerName="manager" Jul 10 17:14:05.698180 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:05.698163 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p" Jul 10 17:14:05.702790 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:05.702769 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vcxl9\"" Jul 10 17:14:05.710518 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:05.710495 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p"] Jul 10 17:14:05.733463 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:05.733413 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ttxx\" (UniqueName: \"kubernetes.io/projected/83610753-2dc9-448f-8763-becc2bc23acf-kube-api-access-7ttxx\") pod \"kuadrant-operator-controller-manager-86878c769f-vzr5p\" (UID: \"83610753-2dc9-448f-8763-becc2bc23acf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p" Jul 10 17:14:05.733463 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:05.733458 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83610753-2dc9-448f-8763-becc2bc23acf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vzr5p\" (UID: \"83610753-2dc9-448f-8763-becc2bc23acf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p" Jul 10 17:14:05.834522 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:05.834484 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ttxx\" (UniqueName: \"kubernetes.io/projected/83610753-2dc9-448f-8763-becc2bc23acf-kube-api-access-7ttxx\") pod \"kuadrant-operator-controller-manager-86878c769f-vzr5p\" (UID: \"83610753-2dc9-448f-8763-becc2bc23acf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p" Jul 10 17:14:05.834522 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:05.834527 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83610753-2dc9-448f-8763-becc2bc23acf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vzr5p\" (UID: \"83610753-2dc9-448f-8763-becc2bc23acf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p" Jul 10 17:14:05.834908 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:05.834890 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83610753-2dc9-448f-8763-becc2bc23acf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vzr5p\" (UID: \"83610753-2dc9-448f-8763-becc2bc23acf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p" Jul 10 17:14:05.845745 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:05.845718 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ttxx\" (UniqueName: \"kubernetes.io/projected/83610753-2dc9-448f-8763-becc2bc23acf-kube-api-access-7ttxx\") pod \"kuadrant-operator-controller-manager-86878c769f-vzr5p\" (UID: \"83610753-2dc9-448f-8763-becc2bc23acf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p" Jul 10 17:14:06.008946 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:06.008863 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p" Jul 10 17:14:06.151946 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:06.151919 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p"] Jul 10 17:14:06.153049 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:14:06.153018 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83610753_2dc9_448f_8763_becc2bc23acf.slice/crio-6b2a751fbe62d2ece70128231390bdca2967166c8ef30a8484daca3846af1344 WatchSource:0}: Error finding container 6b2a751fbe62d2ece70128231390bdca2967166c8ef30a8484daca3846af1344: Status 404 returned error can't find the container with id 6b2a751fbe62d2ece70128231390bdca2967166c8ef30a8484daca3846af1344 Jul 10 17:14:06.240154 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:06.240118 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p" event={"ID":"83610753-2dc9-448f-8763-becc2bc23acf","Type":"ContainerStarted","Data":"32320725ecc348e08e4787c2d521a32bbe512244e1a433a8842e31ec167d7680"} Jul 10 17:14:06.240252 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:06.240181 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p" event={"ID":"83610753-2dc9-448f-8763-becc2bc23acf","Type":"ContainerStarted","Data":"6b2a751fbe62d2ece70128231390bdca2967166c8ef30a8484daca3846af1344"} Jul 10 17:14:06.240326 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:06.240295 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p" Jul 10 17:14:06.272887 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:06.272811 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p" podStartSLOduration=1.27279738 podStartE2EDuration="1.27279738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:14:06.269770503 +0000 UTC m=+1176.157061824" watchObservedRunningTime="2026-07-10 17:14:06.27279738 +0000 UTC m=+1176.160088700" Jul 10 17:14:17.247123 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:17.247093 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vzr5p" Jul 10 17:14:30.594966 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:30.594932 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:14:30.597133 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:14:30.597109 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:15:00.141641 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:15:00.141606 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm"] Jul 10 17:15:00.145175 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:15:00.145151 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm" Jul 10 17:15:00.149369 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:15:00.149349 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kqj5t\"" Jul 10 17:15:00.153452 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:15:00.153424 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm"] Jul 10 17:15:00.157191 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:15:00.157170 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm" Jul 10 17:15:00.301169 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:15:00.301138 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm"] Jul 10 17:15:00.302222 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:15:00.302198 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549c43e0_8ca2_405f_8feb_9bfb2551ec35.slice/crio-99624e36f645d8d9aaf977e5ad86bc569b59d65c1c9a734f24c04a7e93bc7003 WatchSource:0}: Error finding container 99624e36f645d8d9aaf977e5ad86bc569b59d65c1c9a734f24c04a7e93bc7003: Status 404 returned error can't find the container with id 99624e36f645d8d9aaf977e5ad86bc569b59d65c1c9a734f24c04a7e93bc7003 Jul 10 17:15:00.304546 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:15:00.304521 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:15:00.509157 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:15:00.509123 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm" event={"ID":"549c43e0-8ca2-405f-8feb-9bfb2551ec35","Type":"ContainerStarted","Data":"99624e36f645d8d9aaf977e5ad86bc569b59d65c1c9a734f24c04a7e93bc7003"} Jul 10 17:15:04.532515 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:15:04.532472 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm" event={"ID":"549c43e0-8ca2-405f-8feb-9bfb2551ec35","Type":"ContainerStarted","Data":"e4bba7878c35c2c647f6b39c659d829939ad1ba493b641c970df3878202c3e9d"} Jul 10 17:16:00.010155 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:00.010092 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm" podStartSLOduration=56.190140991 podStartE2EDuration="1m0.01007312s" totalImagesPullingTime="3.819932129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:15:04.547381751 +0000 UTC m=+1234.434673095" watchObservedRunningTime="2026-07-10 17:16:00.01007312 +0000 UTC m=+1289.897364440" Jul 10 17:16:00.010748 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:00.010732 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm"] Jul 10 17:16:00.010930 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:00.010912 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm" podUID="549c43e0-8ca2-405f-8feb-9bfb2551ec35" containerName="cleanup" containerID="cri-o://e4bba7878c35c2c647f6b39c659d829939ad1ba493b641c970df3878202c3e9d" gracePeriod=30 Jul 10 17:16:30.199004 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:30.198978 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm" Jul 10 17:16:30.982689 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:30.982654 2567 generic.go:356] "Generic (PLEG): container finished" podID="549c43e0-8ca2-405f-8feb-9bfb2551ec35" containerID="e4bba7878c35c2c647f6b39c659d829939ad1ba493b641c970df3878202c3e9d" exitCode=137 Jul 10 17:16:30.982880 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:30.982722 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm" Jul 10 17:16:30.982880 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:30.982741 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm" event={"ID":"549c43e0-8ca2-405f-8feb-9bfb2551ec35","Type":"ContainerDied","Data":"e4bba7878c35c2c647f6b39c659d829939ad1ba493b641c970df3878202c3e9d"} Jul 10 17:16:30.982880 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:30.982815 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm" event={"ID":"549c43e0-8ca2-405f-8feb-9bfb2551ec35","Type":"ContainerDied","Data":"99624e36f645d8d9aaf977e5ad86bc569b59d65c1c9a734f24c04a7e93bc7003"} Jul 10 17:16:30.982880 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:30.982837 2567 scope.go:117] "RemoveContainer" containerID="e4bba7878c35c2c647f6b39c659d829939ad1ba493b641c970df3878202c3e9d" Jul 10 17:16:30.991581 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:30.991561 2567 scope.go:117] "RemoveContainer" containerID="e4bba7878c35c2c647f6b39c659d829939ad1ba493b641c970df3878202c3e9d" Jul 10 17:16:30.991918 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:16:30.991882 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bba7878c35c2c647f6b39c659d829939ad1ba493b641c970df3878202c3e9d\": container with ID starting with e4bba7878c35c2c647f6b39c659d829939ad1ba493b641c970df3878202c3e9d not found: ID does not exist" containerID="e4bba7878c35c2c647f6b39c659d829939ad1ba493b641c970df3878202c3e9d" Jul 10 17:16:30.991996 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:30.991944 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bba7878c35c2c647f6b39c659d829939ad1ba493b641c970df3878202c3e9d"} err="failed to get container status \"e4bba7878c35c2c647f6b39c659d829939ad1ba493b641c970df3878202c3e9d\": rpc error: code = NotFound desc = could not find container \"e4bba7878c35c2c647f6b39c659d829939ad1ba493b641c970df3878202c3e9d\": container with ID starting with e4bba7878c35c2c647f6b39c659d829939ad1ba493b641c970df3878202c3e9d not found: ID does not exist" Jul 10 17:16:31.017900 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:31.017874 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm"] Jul 10 17:16:31.027220 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:31.027197 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728395-4ppwm"] Jul 10 17:16:32.682460 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:16:32.682423 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="549c43e0-8ca2-405f-8feb-9bfb2551ec35" path="/var/lib/kubelet/pods/549c43e0-8ca2-405f-8feb-9bfb2551ec35/volumes" Jul 10 17:19:30.623261 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:19:30.623226 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:19:30.626182 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:19:30.626156 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:20:43.935108 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:43.935072 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb"] Jul 10 17:20:43.935585 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:43.935502 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549c43e0-8ca2-405f-8feb-9bfb2551ec35" containerName="cleanup" Jul 10 17:20:43.935585 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:43.935518 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="549c43e0-8ca2-405f-8feb-9bfb2551ec35" containerName="cleanup" Jul 10 17:20:43.935585 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:43.935581 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="549c43e0-8ca2-405f-8feb-9bfb2551ec35" containerName="cleanup" Jul 10 17:20:43.938672 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:43.938651 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:43.940854 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:43.940831 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8bab674c-serving-cert\"" Jul 10 17:20:43.941314 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:43.941296 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-maas-api-tls\") pod \"maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb\" (UID: \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:43.941434 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:43.941415 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2l7\" (UniqueName: \"kubernetes.io/projected/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-kube-api-access-js2l7\") pod \"maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb\" (UID: \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:43.941485 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:43.941471 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-openshift-service-ca\") pod \"maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb\" (UID: \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:43.949306 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:43.949260 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb"] Jul 10 17:20:44.042576 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:44.042534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2l7\" (UniqueName: \"kubernetes.io/projected/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-kube-api-access-js2l7\") pod \"maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb\" (UID: \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:44.042576 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:44.042581 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-openshift-service-ca\") pod \"maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb\" (UID: \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:44.042853 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:44.042632 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-maas-api-tls\") pod \"maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb\" (UID: \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:44.042853 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:20:44.042739 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-serving-cert: secret "maas-api-e2e-ait-8bab674c-serving-cert" not found Jul 10 17:20:44.042853 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:20:44.042819 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-maas-api-tls podName:880af6c9-b1a4-4c0d-a808-95e2f0bf09f5 nodeName:}" failed. No retries permitted until 2026-07-10 17:20:44.542793983 +0000 UTC m=+1574.430085281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-maas-api-tls") pod "maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" (UID: "880af6c9-b1a4-4c0d-a808-95e2f0bf09f5") : secret "maas-api-e2e-ait-8bab674c-serving-cert" not found Jul 10 17:20:44.043262 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:44.043240 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-openshift-service-ca\") pod \"maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb\" (UID: \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:44.052292 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:44.052239 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2l7\" (UniqueName: \"kubernetes.io/projected/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-kube-api-access-js2l7\") pod \"maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb\" (UID: \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:44.546749 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:44.546697 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-maas-api-tls\") pod \"maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb\" (UID: \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:44.549396 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:44.549364 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-maas-api-tls\") pod \"maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb\" (UID: \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:44.549535 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:44.549510 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:44.696741 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:44.696715 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb"] Jul 10 17:20:44.699183 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:20:44.699158 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880af6c9_b1a4_4c0d_a808_95e2f0bf09f5.slice/crio-c43b3e5915bd6cf8e70dbc8a263493082e104d5fac6ead8262aac3d89cb86d9c WatchSource:0}: Error finding container c43b3e5915bd6cf8e70dbc8a263493082e104d5fac6ead8262aac3d89cb86d9c: Status 404 returned error can't find the container with id c43b3e5915bd6cf8e70dbc8a263493082e104d5fac6ead8262aac3d89cb86d9c Jul 10 17:20:44.700675 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:44.700650 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:20:45.252065 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:45.252029 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" event={"ID":"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5","Type":"ContainerStarted","Data":"c43b3e5915bd6cf8e70dbc8a263493082e104d5fac6ead8262aac3d89cb86d9c"} Jul 10 17:20:46.258742 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:46.258703 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" event={"ID":"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5","Type":"ContainerStarted","Data":"0c2a0e9b11972b89253b2aec8fdcf7a5d29296e6321deca703bbc9e9d6ffead4"} Jul 10 17:20:46.259147 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:46.258870 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:46.275099 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:46.275037 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" podStartSLOduration=2.564518878 podStartE2EDuration="3.275023353s" totalImagesPullingTime="710.504475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:20:46.274453181 +0000 UTC m=+1576.161744513" watchObservedRunningTime="2026-07-10 17:20:46.275023353 +0000 UTC m=+1576.162314719" Jul 10 17:20:49.270209 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:49.270142 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb"] Jul 10 17:20:49.270802 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:49.270416 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" podUID="880af6c9-b1a4-4c0d-a808-95e2f0bf09f5" containerName="maas-api" containerID="cri-o://0c2a0e9b11972b89253b2aec8fdcf7a5d29296e6321deca703bbc9e9d6ffead4" gracePeriod=30 Jul 10 17:20:49.525756 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:49.525702 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:49.584595 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:49.584553 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-openshift-service-ca\") pod \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\" (UID: \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\") " Jul 10 17:20:49.584781 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:49.584604 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js2l7\" (UniqueName: \"kubernetes.io/projected/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-kube-api-access-js2l7\") pod \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\" (UID: \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\") " Jul 10 17:20:49.584781 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:49.584645 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-maas-api-tls\") pod \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\" (UID: \"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5\") " Jul 10 17:20:49.585010 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:49.584972 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "880af6c9-b1a4-4c0d-a808-95e2f0bf09f5" (UID: "880af6c9-b1a4-4c0d-a808-95e2f0bf09f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:20:49.587024 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:49.586995 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "880af6c9-b1a4-4c0d-a808-95e2f0bf09f5" (UID: "880af6c9-b1a4-4c0d-a808-95e2f0bf09f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:20:49.587192 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:49.587172 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-kube-api-access-js2l7" (OuterVolumeSpecName: "kube-api-access-js2l7") pod "880af6c9-b1a4-4c0d-a808-95e2f0bf09f5" (UID: "880af6c9-b1a4-4c0d-a808-95e2f0bf09f5"). InnerVolumeSpecName "kube-api-access-js2l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:20:49.685484 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:49.685453 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-openshift-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:20:49.685484 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:49.685480 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js2l7\" (UniqueName: \"kubernetes.io/projected/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-kube-api-access-js2l7\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:20:49.685484 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:49.685492 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5-maas-api-tls\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:20:50.280347 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:50.280311 2567 generic.go:356] "Generic (PLEG): container finished" podID="880af6c9-b1a4-4c0d-a808-95e2f0bf09f5" containerID="0c2a0e9b11972b89253b2aec8fdcf7a5d29296e6321deca703bbc9e9d6ffead4" exitCode=0 Jul 10 17:20:50.280816 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:50.280365 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" Jul 10 17:20:50.280816 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:50.280396 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" event={"ID":"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5","Type":"ContainerDied","Data":"0c2a0e9b11972b89253b2aec8fdcf7a5d29296e6321deca703bbc9e9d6ffead4"} Jul 10 17:20:50.280816 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:50.280449 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb" event={"ID":"880af6c9-b1a4-4c0d-a808-95e2f0bf09f5","Type":"ContainerDied","Data":"c43b3e5915bd6cf8e70dbc8a263493082e104d5fac6ead8262aac3d89cb86d9c"} Jul 10 17:20:50.280816 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:50.280487 2567 scope.go:117] "RemoveContainer" containerID="0c2a0e9b11972b89253b2aec8fdcf7a5d29296e6321deca703bbc9e9d6ffead4" Jul 10 17:20:50.290758 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:50.290726 2567 scope.go:117] "RemoveContainer" containerID="0c2a0e9b11972b89253b2aec8fdcf7a5d29296e6321deca703bbc9e9d6ffead4" Jul 10 17:20:50.291047 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:20:50.291025 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2a0e9b11972b89253b2aec8fdcf7a5d29296e6321deca703bbc9e9d6ffead4\": container with ID starting with 0c2a0e9b11972b89253b2aec8fdcf7a5d29296e6321deca703bbc9e9d6ffead4 not found: ID does not exist" containerID="0c2a0e9b11972b89253b2aec8fdcf7a5d29296e6321deca703bbc9e9d6ffead4" Jul 10 17:20:50.291117 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:50.291060 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2a0e9b11972b89253b2aec8fdcf7a5d29296e6321deca703bbc9e9d6ffead4"} err="failed to get container status \"0c2a0e9b11972b89253b2aec8fdcf7a5d29296e6321deca703bbc9e9d6ffead4\": rpc error: code = NotFound desc = could not find container \"0c2a0e9b11972b89253b2aec8fdcf7a5d29296e6321deca703bbc9e9d6ffead4\": container with ID starting with 0c2a0e9b11972b89253b2aec8fdcf7a5d29296e6321deca703bbc9e9d6ffead4 not found: ID does not exist" Jul 10 17:20:50.302036 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:50.302013 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb"] Jul 10 17:20:50.303953 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:50.303934 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bab674c-8559dd68df-4j8rb"] Jul 10 17:20:50.690592 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:50.690556 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="880af6c9-b1a4-4c0d-a808-95e2f0bf09f5" path="/var/lib/kubelet/pods/880af6c9-b1a4-4c0d-a808-95e2f0bf09f5/volumes" Jul 10 17:20:55.486563 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.486510 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d"] Jul 10 17:20:55.487119 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.487098 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="880af6c9-b1a4-4c0d-a808-95e2f0bf09f5" containerName="maas-api" Jul 10 17:20:55.487244 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.487154 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="880af6c9-b1a4-4c0d-a808-95e2f0bf09f5" containerName="maas-api" Jul 10 17:20:55.487354 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.487328 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="880af6c9-b1a4-4c0d-a808-95e2f0bf09f5" containerName="maas-api" Jul 10 17:20:55.491984 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.491961 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.494525 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.494478 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-1f3ba1c6-gw-openshift-default-dockercfg-7l6qh\"" Jul 10 17:20:55.505833 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.505804 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d"] Jul 10 17:20:55.537642 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.537610 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-workload-socket\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.537802 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.537649 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-envoy\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.537802 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.537680 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-credential-socket\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.537802 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.537702 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-data\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.537974 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.537829 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istiod-ca-cert\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.537974 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.537869 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-token\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.537974 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.537935 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-podinfo\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.538114 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.537970 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-workload-certs\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.538114 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.538000 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvtmx\" (UniqueName: \"kubernetes.io/projected/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-kube-api-access-hvtmx\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.639352 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.639309 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istiod-ca-cert\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.639352 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.639354 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-token\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.639611 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.639394 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-podinfo\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.639611 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.639426 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-workload-certs\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.639611 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.639454 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvtmx\" (UniqueName: \"kubernetes.io/projected/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-kube-api-access-hvtmx\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.639611 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.639537 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-workload-socket\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.639611 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.639558 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-envoy\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.639847 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.639629 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-credential-socket\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.639847 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.639684 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-data\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.640104 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.640079 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-data\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.640201 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.640103 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-credential-socket\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.640201 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.640117 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istiod-ca-cert\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.640201 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.640170 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-workload-socket\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.640201 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.640185 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-workload-certs\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.641917 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.641898 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-envoy\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.641996 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.641956 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-podinfo\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.647126 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.647096 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-token\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.647211 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.647197 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvtmx\" (UniqueName: \"kubernetes.io/projected/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-kube-api-access-hvtmx\") pod \"e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.807141 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.807049 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:55.949941 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.949916 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d"] Jul 10 17:20:55.952287 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:20:55.952248 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a67939_b9b2_4fa1_adb4_5a3c5e9cbc37.slice/crio-8d471d6095c616542a2bedffb5d339f791eb531e8265c8080f88c026124400e5 WatchSource:0}: Error finding container 8d471d6095c616542a2bedffb5d339f791eb531e8265c8080f88c026124400e5: Status 404 returned error can't find the container with id 8d471d6095c616542a2bedffb5d339f791eb531e8265c8080f88c026124400e5 Jul 10 17:20:55.954356 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.954319 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:20:55.954447 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.954403 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:20:55.954486 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:55.954450 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:20:56.313332 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:56.313261 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" event={"ID":"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37","Type":"ContainerStarted","Data":"ac5b8fd12e7175f87b90b4b1885efd25c972ea34fc16b05e41435271a7861836"} Jul 10 17:20:56.313544 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:56.313382 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" event={"ID":"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37","Type":"ContainerStarted","Data":"8d471d6095c616542a2bedffb5d339f791eb531e8265c8080f88c026124400e5"} Jul 10 17:20:56.332376 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:56.332317 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" podStartSLOduration=1.332299352 podStartE2EDuration="1.332299352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:20:56.330993536 +0000 UTC m=+1586.218284882" watchObservedRunningTime="2026-07-10 17:20:56.332299352 +0000 UTC m=+1586.219590669" Jul 10 17:20:56.808194 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:56.808155 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:56.841485 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:56.841451 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm"] Jul 10 17:20:56.845033 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:56.845012 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:20:56.847354 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:56.847334 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1f3ba1c6-serving-cert\"" Jul 10 17:20:56.852884 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:56.852860 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm"] Jul 10 17:20:56.953547 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:56.953508 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-openshift-service-ca\") pod \"maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm\" (UID: \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:20:56.953739 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:56.953714 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-maas-api-tls\") pod \"maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm\" (UID: \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:20:56.953785 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:56.953764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gbfz\" (UniqueName: \"kubernetes.io/projected/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-kube-api-access-4gbfz\") pod \"maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm\" (UID: \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:20:57.055244 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:57.055199 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-maas-api-tls\") pod \"maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm\" (UID: \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:20:57.055244 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:57.055248 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gbfz\" (UniqueName: \"kubernetes.io/projected/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-kube-api-access-4gbfz\") pod \"maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm\" (UID: \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:20:57.055535 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:57.055508 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-openshift-service-ca\") pod \"maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm\" (UID: \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:20:57.056176 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:57.056119 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-openshift-service-ca\") pod \"maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm\" (UID: \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:20:57.057885 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:57.057862 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-maas-api-tls\") pod \"maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm\" (UID: \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:20:57.063069 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:57.063012 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gbfz\" (UniqueName: \"kubernetes.io/projected/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-kube-api-access-4gbfz\") pod \"maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm\" (UID: \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:20:57.158116 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:57.158075 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:20:57.301788 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:57.301763 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm"] Jul 10 17:20:57.304172 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:20:57.304128 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cd6fb0_b9a8_4137_b5ad_b74d4ed5b76f.slice/crio-352c4643fb2ddcde4bd03f3142038be803c7818b57b2cc3c2f4af2e9cab3ab5e WatchSource:0}: Error finding container 352c4643fb2ddcde4bd03f3142038be803c7818b57b2cc3c2f4af2e9cab3ab5e: Status 404 returned error can't find the container with id 352c4643fb2ddcde4bd03f3142038be803c7818b57b2cc3c2f4af2e9cab3ab5e Jul 10 17:20:57.321366 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:57.321261 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" event={"ID":"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f","Type":"ContainerStarted","Data":"352c4643fb2ddcde4bd03f3142038be803c7818b57b2cc3c2f4af2e9cab3ab5e"} Jul 10 17:20:57.812192 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:57.812161 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:58.327889 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:58.327853 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" event={"ID":"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f","Type":"ContainerStarted","Data":"de40a3868b1cf8ae13ef77aa739b995f3b2a09884b2467621a6f5a121634d051"} Jul 10 17:20:58.328099 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:58.328018 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:20:58.328183 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:58.328166 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:58.329157 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:58.329135 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:20:58.343089 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:20:58.343039 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" podStartSLOduration=1.957918921 podStartE2EDuration="2.34299964s" totalImagesPullingTime="385.080719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:20:58.341958313 +0000 UTC m=+1588.229249654" watchObservedRunningTime="2026-07-10 17:20:58.34299964 +0000 UTC m=+1588.230290959" Jul 10 17:21:01.679490 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:01.679442 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d"] Jul 10 17:21:01.679983 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:01.679727 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" podUID="20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" containerName="istio-proxy" containerID="cri-o://ac5b8fd12e7175f87b90b4b1885efd25c972ea34fc16b05e41435271a7861836" gracePeriod=30 Jul 10 17:21:01.902297 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:01.899308 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm"] Jul 10 17:21:01.902297 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:01.899561 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" podUID="62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f" containerName="maas-api" containerID="cri-o://de40a3868b1cf8ae13ef77aa739b995f3b2a09884b2467621a6f5a121634d051" gracePeriod=30 Jul 10 17:21:02.211424 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.211397 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:21:02.318562 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.318467 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-maas-api-tls\") pod \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\" (UID: \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\") " Jul 10 17:21:02.318562 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.318554 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-openshift-service-ca\") pod \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\" (UID: \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\") " Jul 10 17:21:02.318758 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.318598 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gbfz\" (UniqueName: \"kubernetes.io/projected/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-kube-api-access-4gbfz\") pod \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\" (UID: \"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f\") " Jul 10 17:21:02.318933 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.318906 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f" (UID: "62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:21:02.320659 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.320635 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f" (UID: "62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:21:02.320797 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.320782 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-kube-api-access-4gbfz" (OuterVolumeSpecName: "kube-api-access-4gbfz") pod "62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f" (UID: "62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f"). InnerVolumeSpecName "kube-api-access-4gbfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:21:02.351695 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.351662 2567 generic.go:356] "Generic (PLEG): container finished" podID="62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f" containerID="de40a3868b1cf8ae13ef77aa739b995f3b2a09884b2467621a6f5a121634d051" exitCode=0 Jul 10 17:21:02.351861 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.351698 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" event={"ID":"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f","Type":"ContainerDied","Data":"de40a3868b1cf8ae13ef77aa739b995f3b2a09884b2467621a6f5a121634d051"} Jul 10 17:21:02.351861 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.351717 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" Jul 10 17:21:02.351861 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.351736 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm" event={"ID":"62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f","Type":"ContainerDied","Data":"352c4643fb2ddcde4bd03f3142038be803c7818b57b2cc3c2f4af2e9cab3ab5e"} Jul 10 17:21:02.351861 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.351760 2567 scope.go:117] "RemoveContainer" containerID="de40a3868b1cf8ae13ef77aa739b995f3b2a09884b2467621a6f5a121634d051" Jul 10 17:21:02.360721 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.360704 2567 scope.go:117] "RemoveContainer" containerID="de40a3868b1cf8ae13ef77aa739b995f3b2a09884b2467621a6f5a121634d051" Jul 10 17:21:02.360966 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:21:02.360950 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de40a3868b1cf8ae13ef77aa739b995f3b2a09884b2467621a6f5a121634d051\": container with ID starting with de40a3868b1cf8ae13ef77aa739b995f3b2a09884b2467621a6f5a121634d051 not found: ID does not exist" containerID="de40a3868b1cf8ae13ef77aa739b995f3b2a09884b2467621a6f5a121634d051" Jul 10 17:21:02.361011 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.360975 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de40a3868b1cf8ae13ef77aa739b995f3b2a09884b2467621a6f5a121634d051"} err="failed to get container status \"de40a3868b1cf8ae13ef77aa739b995f3b2a09884b2467621a6f5a121634d051\": rpc error: code = NotFound desc = could not find container \"de40a3868b1cf8ae13ef77aa739b995f3b2a09884b2467621a6f5a121634d051\": container with ID starting with de40a3868b1cf8ae13ef77aa739b995f3b2a09884b2467621a6f5a121634d051 not found: ID does not exist" Jul 10 17:21:02.371500 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.371472 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm"] Jul 10 17:21:02.375829 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.375800 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f3ba1c6-77fd88d7bb-2bqkm"] Jul 10 17:21:02.420395 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.420359 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gbfz\" (UniqueName: \"kubernetes.io/projected/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-kube-api-access-4gbfz\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:02.420395 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.420391 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-maas-api-tls\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:02.420395 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.420403 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f-openshift-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:02.687579 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:02.687481 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f" path="/var/lib/kubelet/pods/62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f/volumes" Jul 10 17:21:07.851030 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.850999 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:21:07.973409 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.973374 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-workload-socket\") pod \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " Jul 10 17:21:07.973624 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.973428 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-token\") pod \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " Jul 10 17:21:07.973624 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.973459 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istiod-ca-cert\") pod \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " Jul 10 17:21:07.973624 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.973494 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-workload-certs\") pod \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " Jul 10 17:21:07.973624 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.973528 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-podinfo\") pod \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " Jul 10 17:21:07.973862 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.973737 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" (UID: "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:07.973862 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.973821 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvtmx\" (UniqueName: \"kubernetes.io/projected/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-kube-api-access-hvtmx\") pod \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " Jul 10 17:21:07.973976 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.973886 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-credential-socket\") pod \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " Jul 10 17:21:07.973976 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.973931 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-data\") pod \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " Jul 10 17:21:07.973976 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.973959 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-envoy\") pod \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\" (UID: \"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37\") " Jul 10 17:21:07.973976 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.973882 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" (UID: "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:07.974201 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.973943 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" (UID: "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:21:07.974263 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.974224 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" (UID: "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:07.974545 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.974328 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-credential-socket\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:07.974545 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.974347 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-workload-socket\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:07.974545 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.974362 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istiod-ca-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:07.974545 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.974376 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-workload-certs\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:07.974914 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.974886 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-data" (OuterVolumeSpecName: "istio-data") pod "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" (UID: "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:07.976060 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.976028 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-token" (OuterVolumeSpecName: "istio-token") pod "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" (UID: "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:21:07.976518 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.976476 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" (UID: "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:21:07.976969 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.976944 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-kube-api-access-hvtmx" (OuterVolumeSpecName: "kube-api-access-hvtmx") pod "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" (UID: "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37"). InnerVolumeSpecName "kube-api-access-hvtmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:21:07.977065 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:07.976998 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" (UID: "20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:08.075391 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.075352 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-podinfo\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:08.075391 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.075386 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvtmx\" (UniqueName: \"kubernetes.io/projected/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-kube-api-access-hvtmx\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:08.075606 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.075402 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-data\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:08.075606 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.075414 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-envoy\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:08.075606 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.075424 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37-istio-token\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:08.386473 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.386377 2567 generic.go:356] "Generic (PLEG): container finished" podID="20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" containerID="ac5b8fd12e7175f87b90b4b1885efd25c972ea34fc16b05e41435271a7861836" exitCode=0 Jul 10 17:21:08.386473 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.386431 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" Jul 10 17:21:08.386657 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.386466 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" event={"ID":"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37","Type":"ContainerDied","Data":"ac5b8fd12e7175f87b90b4b1885efd25c972ea34fc16b05e41435271a7861836"} Jul 10 17:21:08.386657 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.386525 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d" event={"ID":"20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37","Type":"ContainerDied","Data":"8d471d6095c616542a2bedffb5d339f791eb531e8265c8080f88c026124400e5"} Jul 10 17:21:08.386657 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.386553 2567 scope.go:117] "RemoveContainer" containerID="ac5b8fd12e7175f87b90b4b1885efd25c972ea34fc16b05e41435271a7861836" Jul 10 17:21:08.396828 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.396809 2567 scope.go:117] "RemoveContainer" containerID="ac5b8fd12e7175f87b90b4b1885efd25c972ea34fc16b05e41435271a7861836" Jul 10 17:21:08.397124 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:21:08.397106 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac5b8fd12e7175f87b90b4b1885efd25c972ea34fc16b05e41435271a7861836\": container with ID starting with ac5b8fd12e7175f87b90b4b1885efd25c972ea34fc16b05e41435271a7861836 not found: ID does not exist" containerID="ac5b8fd12e7175f87b90b4b1885efd25c972ea34fc16b05e41435271a7861836" Jul 10 17:21:08.397190 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.397133 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac5b8fd12e7175f87b90b4b1885efd25c972ea34fc16b05e41435271a7861836"} err="failed to get container status \"ac5b8fd12e7175f87b90b4b1885efd25c972ea34fc16b05e41435271a7861836\": rpc error: code = NotFound desc = could not find container \"ac5b8fd12e7175f87b90b4b1885efd25c972ea34fc16b05e41435271a7861836\": container with ID starting with ac5b8fd12e7175f87b90b4b1885efd25c972ea34fc16b05e41435271a7861836 not found: ID does not exist" Jul 10 17:21:08.407761 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.407736 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d"] Jul 10 17:21:08.409869 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.409848 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-1f3ba1c6-gw-openshift-default-75fcbcf455-prt9d"] Jul 10 17:21:08.687467 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.687387 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" path="/var/lib/kubelet/pods/20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37/volumes" Jul 10 17:21:08.947564 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.947534 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw"] Jul 10 17:21:08.948028 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.948006 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f" containerName="maas-api" Jul 10 17:21:08.948201 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.948030 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f" containerName="maas-api" Jul 10 17:21:08.948201 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.948053 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" containerName="istio-proxy" Jul 10 17:21:08.948201 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.948061 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" containerName="istio-proxy" Jul 10 17:21:08.948201 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.948181 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a67939-b9b2-4fa1-adb4-5a3c5e9cbc37" containerName="istio-proxy" Jul 10 17:21:08.948201 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.948202 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="62cd6fb0-b9a8-4137-b5ad-b74d4ed5b76f" containerName="maas-api" Jul 10 17:21:08.951108 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.951090 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:08.953428 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.953408 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-45063052-serving-cert\"" Jul 10 17:21:08.962190 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:08.962145 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw"] Jul 10 17:21:09.086728 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:09.086692 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f210de-d217-4887-874a-853f9eab8d04-maas-api-tls\") pod \"maas-api-e2e-ait-45063052-6875c75bc-ddwkw\" (UID: \"06f210de-d217-4887-874a-853f9eab8d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:09.086728 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:09.086730 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f210de-d217-4887-874a-853f9eab8d04-openshift-service-ca\") pod \"maas-api-e2e-ait-45063052-6875c75bc-ddwkw\" (UID: \"06f210de-d217-4887-874a-853f9eab8d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:09.086994 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:09.086751 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84j2\" (UniqueName: \"kubernetes.io/projected/06f210de-d217-4887-874a-853f9eab8d04-kube-api-access-l84j2\") pod \"maas-api-e2e-ait-45063052-6875c75bc-ddwkw\" (UID: \"06f210de-d217-4887-874a-853f9eab8d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:09.187794 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:09.187747 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f210de-d217-4887-874a-853f9eab8d04-maas-api-tls\") pod \"maas-api-e2e-ait-45063052-6875c75bc-ddwkw\" (UID: \"06f210de-d217-4887-874a-853f9eab8d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:09.188004 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:09.187796 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f210de-d217-4887-874a-853f9eab8d04-openshift-service-ca\") pod \"maas-api-e2e-ait-45063052-6875c75bc-ddwkw\" (UID: \"06f210de-d217-4887-874a-853f9eab8d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:09.188004 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:09.187842 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84j2\" (UniqueName: \"kubernetes.io/projected/06f210de-d217-4887-874a-853f9eab8d04-kube-api-access-l84j2\") pod \"maas-api-e2e-ait-45063052-6875c75bc-ddwkw\" (UID: \"06f210de-d217-4887-874a-853f9eab8d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:09.188494 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:09.188469 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f210de-d217-4887-874a-853f9eab8d04-openshift-service-ca\") pod \"maas-api-e2e-ait-45063052-6875c75bc-ddwkw\" (UID: \"06f210de-d217-4887-874a-853f9eab8d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:09.190647 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:09.190623 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f210de-d217-4887-874a-853f9eab8d04-maas-api-tls\") pod \"maas-api-e2e-ait-45063052-6875c75bc-ddwkw\" (UID: \"06f210de-d217-4887-874a-853f9eab8d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:09.195488 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:09.195466 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84j2\" (UniqueName: \"kubernetes.io/projected/06f210de-d217-4887-874a-853f9eab8d04-kube-api-access-l84j2\") pod \"maas-api-e2e-ait-45063052-6875c75bc-ddwkw\" (UID: \"06f210de-d217-4887-874a-853f9eab8d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:09.262964 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:09.262889 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:09.419463 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:09.419436 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw"] Jul 10 17:21:09.420974 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:21:09.420943 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f210de_d217_4887_874a_853f9eab8d04.slice/crio-3c6b6b665e1211a53e7e5c85ee7dd6614751c28bfe3019ea5b501257b60064d6 WatchSource:0}: Error finding container 3c6b6b665e1211a53e7e5c85ee7dd6614751c28bfe3019ea5b501257b60064d6: Status 404 returned error can't find the container with id 3c6b6b665e1211a53e7e5c85ee7dd6614751c28bfe3019ea5b501257b60064d6 Jul 10 17:21:10.400619 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:10.400578 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" event={"ID":"06f210de-d217-4887-874a-853f9eab8d04","Type":"ContainerStarted","Data":"3c6b6b665e1211a53e7e5c85ee7dd6614751c28bfe3019ea5b501257b60064d6"} Jul 10 17:21:11.407212 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:11.407171 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" event={"ID":"06f210de-d217-4887-874a-853f9eab8d04","Type":"ContainerStarted","Data":"63e5f2f88a713fbb9b17bc2fa768a6a5cd3ab7ceb6a85bd658416d2fd3ea7b8b"} Jul 10 17:21:11.407631 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:11.407306 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:11.423451 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:11.423404 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" podStartSLOduration=2.312875343 podStartE2EDuration="3.423387527s" totalImagesPullingTime="1.110512184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:21:11.421234921 +0000 UTC m=+1601.308526242" watchObservedRunningTime="2026-07-10 17:21:11.423387527 +0000 UTC m=+1601.310678847" Jul 10 17:21:14.529446 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:14.529411 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw"] Jul 10 17:21:14.529947 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:14.529618 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" podUID="06f210de-d217-4887-874a-853f9eab8d04" containerName="maas-api" containerID="cri-o://63e5f2f88a713fbb9b17bc2fa768a6a5cd3ab7ceb6a85bd658416d2fd3ea7b8b" gracePeriod=30 Jul 10 17:21:14.776551 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:14.776522 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:14.943345 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:14.943317 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f210de-d217-4887-874a-853f9eab8d04-maas-api-tls\") pod \"06f210de-d217-4887-874a-853f9eab8d04\" (UID: \"06f210de-d217-4887-874a-853f9eab8d04\") " Jul 10 17:21:14.943527 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:14.943351 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l84j2\" (UniqueName: \"kubernetes.io/projected/06f210de-d217-4887-874a-853f9eab8d04-kube-api-access-l84j2\") pod \"06f210de-d217-4887-874a-853f9eab8d04\" (UID: \"06f210de-d217-4887-874a-853f9eab8d04\") " Jul 10 17:21:14.943527 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:14.943421 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f210de-d217-4887-874a-853f9eab8d04-openshift-service-ca\") pod \"06f210de-d217-4887-874a-853f9eab8d04\" (UID: \"06f210de-d217-4887-874a-853f9eab8d04\") " Jul 10 17:21:14.943789 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:14.943757 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f210de-d217-4887-874a-853f9eab8d04-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06f210de-d217-4887-874a-853f9eab8d04" (UID: "06f210de-d217-4887-874a-853f9eab8d04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:21:14.945566 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:14.945542 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f210de-d217-4887-874a-853f9eab8d04-kube-api-access-l84j2" (OuterVolumeSpecName: "kube-api-access-l84j2") pod "06f210de-d217-4887-874a-853f9eab8d04" (UID: "06f210de-d217-4887-874a-853f9eab8d04"). InnerVolumeSpecName "kube-api-access-l84j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:21:14.945683 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:14.945664 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f210de-d217-4887-874a-853f9eab8d04-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06f210de-d217-4887-874a-853f9eab8d04" (UID: "06f210de-d217-4887-874a-853f9eab8d04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:21:15.044492 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:15.044444 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f210de-d217-4887-874a-853f9eab8d04-maas-api-tls\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:15.044492 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:15.044486 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l84j2\" (UniqueName: \"kubernetes.io/projected/06f210de-d217-4887-874a-853f9eab8d04-kube-api-access-l84j2\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:15.044492 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:15.044497 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f210de-d217-4887-874a-853f9eab8d04-openshift-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:15.427816 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:15.427725 2567 generic.go:356] "Generic (PLEG): container finished" podID="06f210de-d217-4887-874a-853f9eab8d04" containerID="63e5f2f88a713fbb9b17bc2fa768a6a5cd3ab7ceb6a85bd658416d2fd3ea7b8b" exitCode=0 Jul 10 17:21:15.427816 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:15.427771 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" Jul 10 17:21:15.428002 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:15.427813 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" event={"ID":"06f210de-d217-4887-874a-853f9eab8d04","Type":"ContainerDied","Data":"63e5f2f88a713fbb9b17bc2fa768a6a5cd3ab7ceb6a85bd658416d2fd3ea7b8b"} Jul 10 17:21:15.428002 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:15.427863 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw" event={"ID":"06f210de-d217-4887-874a-853f9eab8d04","Type":"ContainerDied","Data":"3c6b6b665e1211a53e7e5c85ee7dd6614751c28bfe3019ea5b501257b60064d6"} Jul 10 17:21:15.428002 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:15.427890 2567 scope.go:117] "RemoveContainer" containerID="63e5f2f88a713fbb9b17bc2fa768a6a5cd3ab7ceb6a85bd658416d2fd3ea7b8b" Jul 10 17:21:15.437519 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:15.437500 2567 scope.go:117] "RemoveContainer" containerID="63e5f2f88a713fbb9b17bc2fa768a6a5cd3ab7ceb6a85bd658416d2fd3ea7b8b" Jul 10 17:21:15.437802 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:21:15.437784 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e5f2f88a713fbb9b17bc2fa768a6a5cd3ab7ceb6a85bd658416d2fd3ea7b8b\": container with ID starting with 63e5f2f88a713fbb9b17bc2fa768a6a5cd3ab7ceb6a85bd658416d2fd3ea7b8b not found: ID does not exist" containerID="63e5f2f88a713fbb9b17bc2fa768a6a5cd3ab7ceb6a85bd658416d2fd3ea7b8b" Jul 10 17:21:15.437851 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:15.437811 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e5f2f88a713fbb9b17bc2fa768a6a5cd3ab7ceb6a85bd658416d2fd3ea7b8b"} err="failed to get container status \"63e5f2f88a713fbb9b17bc2fa768a6a5cd3ab7ceb6a85bd658416d2fd3ea7b8b\": rpc error: code = NotFound desc = could not find container \"63e5f2f88a713fbb9b17bc2fa768a6a5cd3ab7ceb6a85bd658416d2fd3ea7b8b\": container with ID starting with 63e5f2f88a713fbb9b17bc2fa768a6a5cd3ab7ceb6a85bd658416d2fd3ea7b8b not found: ID does not exist" Jul 10 17:21:15.447944 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:15.447920 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw"] Jul 10 17:21:15.453137 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:15.453116 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45063052-6875c75bc-ddwkw"] Jul 10 17:21:16.683156 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:16.683122 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06f210de-d217-4887-874a-853f9eab8d04" path="/var/lib/kubelet/pods/06f210de-d217-4887-874a-853f9eab8d04/volumes" Jul 10 17:21:27.642188 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.642150 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm"] Jul 10 17:21:27.642684 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.642551 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f210de-d217-4887-874a-853f9eab8d04" containerName="maas-api" Jul 10 17:21:27.642684 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.642563 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f210de-d217-4887-874a-853f9eab8d04" containerName="maas-api" Jul 10 17:21:27.642684 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.642642 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f210de-d217-4887-874a-853f9eab8d04" containerName="maas-api" Jul 10 17:21:27.647249 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.647229 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:27.649768 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.649745 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-7dfb6652-serving-cert\"" Jul 10 17:21:27.656978 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.656955 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm"] Jul 10 17:21:27.779775 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.779742 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5s82\" (UniqueName: \"kubernetes.io/projected/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-kube-api-access-g5s82\") pod \"maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm\" (UID: \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:27.779943 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.779796 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-maas-api-tls\") pod \"maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm\" (UID: \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:27.779943 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.779814 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-openshift-service-ca\") pod \"maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm\" (UID: \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:27.880981 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.880942 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-maas-api-tls\") pod \"maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm\" (UID: \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:27.881137 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.880989 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-openshift-service-ca\") pod \"maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm\" (UID: \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:27.881137 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.881096 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5s82\" (UniqueName: \"kubernetes.io/projected/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-kube-api-access-g5s82\") pod \"maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm\" (UID: \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:27.881137 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:21:27.881120 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-serving-cert: secret "maas-api-e2e-derive-7dfb6652-serving-cert" not found Jul 10 17:21:27.881357 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:21:27.881223 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-maas-api-tls podName:7c31ecd5-3f57-477b-aefd-f1f2fd0034ec nodeName:}" failed. No retries permitted until 2026-07-10 17:21:28.38120392 +0000 UTC m=+1618.268495220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-maas-api-tls") pod "maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" (UID: "7c31ecd5-3f57-477b-aefd-f1f2fd0034ec") : secret "maas-api-e2e-derive-7dfb6652-serving-cert" not found Jul 10 17:21:27.881774 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.881754 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-openshift-service-ca\") pod \"maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm\" (UID: \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:27.890199 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:27.890173 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5s82\" (UniqueName: \"kubernetes.io/projected/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-kube-api-access-g5s82\") pod \"maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm\" (UID: \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:28.387074 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:28.387035 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-maas-api-tls\") pod \"maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm\" (UID: \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:28.389595 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:28.389569 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-maas-api-tls\") pod \"maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm\" (UID: \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:28.558562 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:28.558512 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:28.701311 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:28.701255 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm"] Jul 10 17:21:28.703644 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:21:28.703581 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c31ecd5_3f57_477b_aefd_f1f2fd0034ec.slice/crio-893a38126cf41ccedc0dad03354520761d1829a427e6c1fc25783a670834d14c WatchSource:0}: Error finding container 893a38126cf41ccedc0dad03354520761d1829a427e6c1fc25783a670834d14c: Status 404 returned error can't find the container with id 893a38126cf41ccedc0dad03354520761d1829a427e6c1fc25783a670834d14c Jul 10 17:21:29.508328 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:29.508258 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" event={"ID":"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec","Type":"ContainerStarted","Data":"897b057d4986264c8fd9488e45547b1556b5721caffea89ad96a8adebfe33f5b"} Jul 10 17:21:29.508522 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:29.508401 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:29.508522 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:29.508426 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" event={"ID":"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec","Type":"ContainerStarted","Data":"893a38126cf41ccedc0dad03354520761d1829a427e6c1fc25783a670834d14c"} Jul 10 17:21:29.525254 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:29.525200 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" podStartSLOduration=2.125655948 podStartE2EDuration="2.525183597s" totalImagesPullingTime="399.527649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:21:29.524128593 +0000 UTC m=+1619.411419914" watchObservedRunningTime="2026-07-10 17:21:29.525183597 +0000 UTC m=+1619.412474917" Jul 10 17:21:32.575480 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:32.575415 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm"] Jul 10 17:21:32.575969 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:32.575642 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" podUID="7c31ecd5-3f57-477b-aefd-f1f2fd0034ec" containerName="maas-api" containerID="cri-o://897b057d4986264c8fd9488e45547b1556b5721caffea89ad96a8adebfe33f5b" gracePeriod=30 Jul 10 17:21:32.832507 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:32.832452 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:32.932853 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:32.932819 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-openshift-service-ca\") pod \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\" (UID: \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\") " Jul 10 17:21:32.933038 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:32.932871 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-maas-api-tls\") pod \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\" (UID: \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\") " Jul 10 17:21:32.933038 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:32.932897 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5s82\" (UniqueName: \"kubernetes.io/projected/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-kube-api-access-g5s82\") pod \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\" (UID: \"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec\") " Jul 10 17:21:32.933298 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:32.933242 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c31ecd5-3f57-477b-aefd-f1f2fd0034ec" (UID: "7c31ecd5-3f57-477b-aefd-f1f2fd0034ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:21:32.935170 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:32.935144 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-kube-api-access-g5s82" (OuterVolumeSpecName: "kube-api-access-g5s82") pod "7c31ecd5-3f57-477b-aefd-f1f2fd0034ec" (UID: "7c31ecd5-3f57-477b-aefd-f1f2fd0034ec"). InnerVolumeSpecName "kube-api-access-g5s82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:21:32.935323 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:32.935253 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c31ecd5-3f57-477b-aefd-f1f2fd0034ec" (UID: "7c31ecd5-3f57-477b-aefd-f1f2fd0034ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:21:33.033884 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:33.033846 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5s82\" (UniqueName: \"kubernetes.io/projected/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-kube-api-access-g5s82\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:33.033884 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:33.033879 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-openshift-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:33.033884 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:33.033890 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec-maas-api-tls\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:21:33.529667 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:33.529625 2567 generic.go:356] "Generic (PLEG): container finished" podID="7c31ecd5-3f57-477b-aefd-f1f2fd0034ec" containerID="897b057d4986264c8fd9488e45547b1556b5721caffea89ad96a8adebfe33f5b" exitCode=0 Jul 10 17:21:33.529862 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:33.529692 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" Jul 10 17:21:33.529862 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:33.529712 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" event={"ID":"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec","Type":"ContainerDied","Data":"897b057d4986264c8fd9488e45547b1556b5721caffea89ad96a8adebfe33f5b"} Jul 10 17:21:33.529862 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:33.529795 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm" event={"ID":"7c31ecd5-3f57-477b-aefd-f1f2fd0034ec","Type":"ContainerDied","Data":"893a38126cf41ccedc0dad03354520761d1829a427e6c1fc25783a670834d14c"} Jul 10 17:21:33.529862 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:33.529818 2567 scope.go:117] "RemoveContainer" containerID="897b057d4986264c8fd9488e45547b1556b5721caffea89ad96a8adebfe33f5b" Jul 10 17:21:33.539246 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:33.539224 2567 scope.go:117] "RemoveContainer" containerID="897b057d4986264c8fd9488e45547b1556b5721caffea89ad96a8adebfe33f5b" Jul 10 17:21:33.539540 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:21:33.539518 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897b057d4986264c8fd9488e45547b1556b5721caffea89ad96a8adebfe33f5b\": container with ID starting with 897b057d4986264c8fd9488e45547b1556b5721caffea89ad96a8adebfe33f5b not found: ID does not exist" containerID="897b057d4986264c8fd9488e45547b1556b5721caffea89ad96a8adebfe33f5b" Jul 10 17:21:33.539595 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:33.539549 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897b057d4986264c8fd9488e45547b1556b5721caffea89ad96a8adebfe33f5b"} err="failed to get container status \"897b057d4986264c8fd9488e45547b1556b5721caffea89ad96a8adebfe33f5b\": rpc error: code = NotFound desc = could not find container \"897b057d4986264c8fd9488e45547b1556b5721caffea89ad96a8adebfe33f5b\": container with ID starting with 897b057d4986264c8fd9488e45547b1556b5721caffea89ad96a8adebfe33f5b not found: ID does not exist" Jul 10 17:21:33.548812 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:33.548791 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm"] Jul 10 17:21:33.551627 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:33.551606 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7dfb6652-65d5f459f-z2fxm"] Jul 10 17:21:34.681734 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:21:34.681701 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c31ecd5-3f57-477b-aefd-f1f2fd0034ec" path="/var/lib/kubelet/pods/7c31ecd5-3f57-477b-aefd-f1f2fd0034ec/volumes" Jul 10 17:23:18.748572 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.748485 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g"] Jul 10 17:23:18.749169 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.749091 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c31ecd5-3f57-477b-aefd-f1f2fd0034ec" containerName="maas-api" Jul 10 17:23:18.749169 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.749111 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c31ecd5-3f57-477b-aefd-f1f2fd0034ec" containerName="maas-api" Jul 10 17:23:18.749344 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.749207 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c31ecd5-3f57-477b-aefd-f1f2fd0034ec" containerName="maas-api" Jul 10 17:23:18.752728 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.752705 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:18.755549 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.755525 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3ff7ed0c-serving-cert\"" Jul 10 17:23:18.766916 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.766891 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g"] Jul 10 17:23:18.812586 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.812550 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/752239dd-f0f7-40bb-868c-20caf1138dd3-maas-api-tls\") pod \"maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g\" (UID: \"752239dd-f0f7-40bb-868c-20caf1138dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:18.812586 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.812588 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/752239dd-f0f7-40bb-868c-20caf1138dd3-openshift-service-ca\") pod \"maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g\" (UID: \"752239dd-f0f7-40bb-868c-20caf1138dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:18.812800 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.812660 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52hcs\" (UniqueName: \"kubernetes.io/projected/752239dd-f0f7-40bb-868c-20caf1138dd3-kube-api-access-52hcs\") pod \"maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g\" (UID: \"752239dd-f0f7-40bb-868c-20caf1138dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:18.913802 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.913765 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/752239dd-f0f7-40bb-868c-20caf1138dd3-maas-api-tls\") pod \"maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g\" (UID: \"752239dd-f0f7-40bb-868c-20caf1138dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:18.913802 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.913803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/752239dd-f0f7-40bb-868c-20caf1138dd3-openshift-service-ca\") pod \"maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g\" (UID: \"752239dd-f0f7-40bb-868c-20caf1138dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:18.914061 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:23:18.913926 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-serving-cert: secret "maas-api-e2e-mt-3ff7ed0c-serving-cert" not found Jul 10 17:23:18.914061 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.913937 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52hcs\" (UniqueName: \"kubernetes.io/projected/752239dd-f0f7-40bb-868c-20caf1138dd3-kube-api-access-52hcs\") pod \"maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g\" (UID: \"752239dd-f0f7-40bb-868c-20caf1138dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:18.914061 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:23:18.913993 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752239dd-f0f7-40bb-868c-20caf1138dd3-maas-api-tls podName:752239dd-f0f7-40bb-868c-20caf1138dd3 nodeName:}" failed. No retries permitted until 2026-07-10 17:23:19.413976672 +0000 UTC m=+1729.301267971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/752239dd-f0f7-40bb-868c-20caf1138dd3-maas-api-tls") pod "maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" (UID: "752239dd-f0f7-40bb-868c-20caf1138dd3") : secret "maas-api-e2e-mt-3ff7ed0c-serving-cert" not found Jul 10 17:23:18.914533 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.914514 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/752239dd-f0f7-40bb-868c-20caf1138dd3-openshift-service-ca\") pod \"maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g\" (UID: \"752239dd-f0f7-40bb-868c-20caf1138dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:18.923955 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:18.923922 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52hcs\" (UniqueName: \"kubernetes.io/projected/752239dd-f0f7-40bb-868c-20caf1138dd3-kube-api-access-52hcs\") pod \"maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g\" (UID: \"752239dd-f0f7-40bb-868c-20caf1138dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:19.419200 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:19.419161 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/752239dd-f0f7-40bb-868c-20caf1138dd3-maas-api-tls\") pod \"maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g\" (UID: \"752239dd-f0f7-40bb-868c-20caf1138dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:19.421880 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:19.421846 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/752239dd-f0f7-40bb-868c-20caf1138dd3-maas-api-tls\") pod \"maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g\" (UID: \"752239dd-f0f7-40bb-868c-20caf1138dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:19.663675 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:19.663632 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:19.813343 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:19.813294 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g"] Jul 10 17:23:19.816500 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:23:19.816467 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752239dd_f0f7_40bb_868c_20caf1138dd3.slice/crio-6a9b1eb532b47b2a46893a1d6b9eac0d9534288e523c9ea3477f9f7b773bd315 WatchSource:0}: Error finding container 6a9b1eb532b47b2a46893a1d6b9eac0d9534288e523c9ea3477f9f7b773bd315: Status 404 returned error can't find the container with id 6a9b1eb532b47b2a46893a1d6b9eac0d9534288e523c9ea3477f9f7b773bd315 Jul 10 17:23:20.052770 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:20.052727 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" event={"ID":"752239dd-f0f7-40bb-868c-20caf1138dd3","Type":"ContainerStarted","Data":"6a9b1eb532b47b2a46893a1d6b9eac0d9534288e523c9ea3477f9f7b773bd315"} Jul 10 17:23:21.058446 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:21.058405 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" event={"ID":"752239dd-f0f7-40bb-868c-20caf1138dd3","Type":"ContainerStarted","Data":"d75941ee0fa6b7ff2c00ba9ef6248d296c7d3d2f8470acb6eef37d5b4b55a162"} Jul 10 17:23:21.058846 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:21.058518 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:27.069427 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:27.069392 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:27.087026 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:27.086969 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" podStartSLOduration=8.678049336 podStartE2EDuration="9.086954197s" totalImagesPullingTime="408.904861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:23:21.078762542 +0000 UTC m=+1730.966053863" watchObservedRunningTime="2026-07-10 17:23:27.086954197 +0000 UTC m=+1736.974245564" Jul 10 17:23:30.884769 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:30.884727 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g"] Jul 10 17:23:30.885255 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:30.884981 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" podUID="752239dd-f0f7-40bb-868c-20caf1138dd3" containerName="maas-api" containerID="cri-o://d75941ee0fa6b7ff2c00ba9ef6248d296c7d3d2f8470acb6eef37d5b4b55a162" gracePeriod=30 Jul 10 17:23:31.109914 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:31.109874 2567 generic.go:356] "Generic (PLEG): container finished" podID="752239dd-f0f7-40bb-868c-20caf1138dd3" containerID="d75941ee0fa6b7ff2c00ba9ef6248d296c7d3d2f8470acb6eef37d5b4b55a162" exitCode=0 Jul 10 17:23:31.110080 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:31.109940 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" event={"ID":"752239dd-f0f7-40bb-868c-20caf1138dd3","Type":"ContainerDied","Data":"d75941ee0fa6b7ff2c00ba9ef6248d296c7d3d2f8470acb6eef37d5b4b55a162"} Jul 10 17:23:31.148554 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:31.148487 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:31.239523 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:31.239484 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52hcs\" (UniqueName: \"kubernetes.io/projected/752239dd-f0f7-40bb-868c-20caf1138dd3-kube-api-access-52hcs\") pod \"752239dd-f0f7-40bb-868c-20caf1138dd3\" (UID: \"752239dd-f0f7-40bb-868c-20caf1138dd3\") " Jul 10 17:23:31.239700 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:31.239564 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/752239dd-f0f7-40bb-868c-20caf1138dd3-maas-api-tls\") pod \"752239dd-f0f7-40bb-868c-20caf1138dd3\" (UID: \"752239dd-f0f7-40bb-868c-20caf1138dd3\") " Jul 10 17:23:31.239700 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:31.239597 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/752239dd-f0f7-40bb-868c-20caf1138dd3-openshift-service-ca\") pod \"752239dd-f0f7-40bb-868c-20caf1138dd3\" (UID: \"752239dd-f0f7-40bb-868c-20caf1138dd3\") " Jul 10 17:23:31.239981 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:31.239951 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752239dd-f0f7-40bb-868c-20caf1138dd3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "752239dd-f0f7-40bb-868c-20caf1138dd3" (UID: "752239dd-f0f7-40bb-868c-20caf1138dd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:23:31.241807 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:31.241777 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/752239dd-f0f7-40bb-868c-20caf1138dd3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "752239dd-f0f7-40bb-868c-20caf1138dd3" (UID: "752239dd-f0f7-40bb-868c-20caf1138dd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:23:31.241807 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:31.241779 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752239dd-f0f7-40bb-868c-20caf1138dd3-kube-api-access-52hcs" (OuterVolumeSpecName: "kube-api-access-52hcs") pod "752239dd-f0f7-40bb-868c-20caf1138dd3" (UID: "752239dd-f0f7-40bb-868c-20caf1138dd3"). InnerVolumeSpecName "kube-api-access-52hcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:23:31.340603 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:31.340557 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52hcs\" (UniqueName: \"kubernetes.io/projected/752239dd-f0f7-40bb-868c-20caf1138dd3-kube-api-access-52hcs\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:23:31.340603 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:31.340597 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/752239dd-f0f7-40bb-868c-20caf1138dd3-maas-api-tls\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:23:31.340603 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:31.340609 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/752239dd-f0f7-40bb-868c-20caf1138dd3-openshift-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:23:32.116942 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:32.116910 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" event={"ID":"752239dd-f0f7-40bb-868c-20caf1138dd3","Type":"ContainerDied","Data":"6a9b1eb532b47b2a46893a1d6b9eac0d9534288e523c9ea3477f9f7b773bd315"} Jul 10 17:23:32.117371 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:32.116974 2567 scope.go:117] "RemoveContainer" containerID="d75941ee0fa6b7ff2c00ba9ef6248d296c7d3d2f8470acb6eef37d5b4b55a162" Jul 10 17:23:32.117371 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:32.116919 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g" Jul 10 17:23:32.137209 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:32.137183 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g"] Jul 10 17:23:32.142865 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:32.142841 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ff7ed0c-7865bb86cf-7lz2g"] Jul 10 17:23:32.683092 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:23:32.683056 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="752239dd-f0f7-40bb-868c-20caf1138dd3" path="/var/lib/kubelet/pods/752239dd-f0f7-40bb-868c-20caf1138dd3/volumes" Jul 10 17:24:01.962546 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:01.962510 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq"] Jul 10 17:24:01.963263 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:01.962912 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="752239dd-f0f7-40bb-868c-20caf1138dd3" containerName="maas-api" Jul 10 17:24:01.963263 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:01.962925 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="752239dd-f0f7-40bb-868c-20caf1138dd3" containerName="maas-api" Jul 10 17:24:01.963263 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:01.962986 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="752239dd-f0f7-40bb-868c-20caf1138dd3" containerName="maas-api" Jul 10 17:24:01.966145 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:01.966128 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:01.968526 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:01.968503 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d9b2224f-serving-cert\"" Jul 10 17:24:01.983533 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:01.983506 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq"] Jul 10 17:24:02.019051 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:02.019020 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-openshift-service-ca\") pod \"maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq\" (UID: \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:02.019231 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:02.019076 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-maas-api-tls\") pod \"maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq\" (UID: \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:02.019231 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:02.019148 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxl8\" (UniqueName: \"kubernetes.io/projected/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-kube-api-access-drxl8\") pod \"maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq\" (UID: \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:02.119670 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:02.119632 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-maas-api-tls\") pod \"maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq\" (UID: \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:02.119843 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:02.119695 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxl8\" (UniqueName: \"kubernetes.io/projected/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-kube-api-access-drxl8\") pod \"maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq\" (UID: \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:02.119843 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:24:02.119782 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-serving-cert: secret "maas-api-e2e-mt-d9b2224f-serving-cert" not found Jul 10 17:24:02.119940 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:02.119783 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-openshift-service-ca\") pod \"maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq\" (UID: \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:02.119940 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:24:02.119851 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-maas-api-tls podName:8adb927c-ab23-47e3-b3eb-04e3e77e9d87 nodeName:}" failed. No retries permitted until 2026-07-10 17:24:02.619830417 +0000 UTC m=+1772.507121716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-maas-api-tls") pod "maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" (UID: "8adb927c-ab23-47e3-b3eb-04e3e77e9d87") : secret "maas-api-e2e-mt-d9b2224f-serving-cert" not found Jul 10 17:24:02.120364 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:02.120346 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-openshift-service-ca\") pod \"maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq\" (UID: \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:02.134325 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:02.134294 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxl8\" (UniqueName: \"kubernetes.io/projected/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-kube-api-access-drxl8\") pod \"maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq\" (UID: \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:02.623892 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:02.623845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-maas-api-tls\") pod \"maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq\" (UID: \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:02.626647 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:02.626612 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-maas-api-tls\") pod \"maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq\" (UID: \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:02.876934 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:02.876825 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:03.049658 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:03.049614 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq"] Jul 10 17:24:03.052091 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:24:03.052049 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8adb927c_ab23_47e3_b3eb_04e3e77e9d87.slice/crio-a191a175eeab849814555af522a0781b519a261bc048366203fd7e53fcee8f22 WatchSource:0}: Error finding container a191a175eeab849814555af522a0781b519a261bc048366203fd7e53fcee8f22: Status 404 returned error can't find the container with id a191a175eeab849814555af522a0781b519a261bc048366203fd7e53fcee8f22 Jul 10 17:24:03.279136 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:03.279094 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" event={"ID":"8adb927c-ab23-47e3-b3eb-04e3e77e9d87","Type":"ContainerStarted","Data":"a191a175eeab849814555af522a0781b519a261bc048366203fd7e53fcee8f22"} Jul 10 17:24:04.285253 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:04.285217 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" event={"ID":"8adb927c-ab23-47e3-b3eb-04e3e77e9d87","Type":"ContainerStarted","Data":"84f325f06583f4e6c1908dbb1d9eb2f6c6bd86d37edc20c729d75a056674482a"} Jul 10 17:24:04.285676 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:04.285333 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:04.305389 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:04.305330 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" podStartSLOduration=2.924296326 podStartE2EDuration="3.305313755s" totalImagesPullingTime="381.017429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:24:04.304378547 +0000 UTC m=+1774.191669879" watchObservedRunningTime="2026-07-10 17:24:04.305313755 +0000 UTC m=+1774.192605074" Jul 10 17:24:10.296080 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:10.296049 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:30.652907 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:30.652870 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:24:30.655882 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:30.655861 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:24:44.351531 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.351492 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq"] Jul 10 17:24:44.352061 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.351749 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" podUID="8adb927c-ab23-47e3-b3eb-04e3e77e9d87" containerName="maas-api" containerID="cri-o://84f325f06583f4e6c1908dbb1d9eb2f6c6bd86d37edc20c729d75a056674482a" gracePeriod=30 Jul 10 17:24:44.481693 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.481658 2567 generic.go:356] "Generic (PLEG): container finished" podID="8adb927c-ab23-47e3-b3eb-04e3e77e9d87" containerID="84f325f06583f4e6c1908dbb1d9eb2f6c6bd86d37edc20c729d75a056674482a" exitCode=0 Jul 10 17:24:44.481895 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.481771 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" event={"ID":"8adb927c-ab23-47e3-b3eb-04e3e77e9d87","Type":"ContainerDied","Data":"84f325f06583f4e6c1908dbb1d9eb2f6c6bd86d37edc20c729d75a056674482a"} Jul 10 17:24:44.603240 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.603175 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:44.720419 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.720387 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drxl8\" (UniqueName: \"kubernetes.io/projected/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-kube-api-access-drxl8\") pod \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\" (UID: \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\") " Jul 10 17:24:44.720604 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.720527 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-maas-api-tls\") pod \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\" (UID: \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\") " Jul 10 17:24:44.720604 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.720563 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-openshift-service-ca\") pod \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\" (UID: \"8adb927c-ab23-47e3-b3eb-04e3e77e9d87\") " Jul 10 17:24:44.720960 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.720933 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8adb927c-ab23-47e3-b3eb-04e3e77e9d87" (UID: "8adb927c-ab23-47e3-b3eb-04e3e77e9d87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:24:44.722764 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.722732 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8adb927c-ab23-47e3-b3eb-04e3e77e9d87" (UID: "8adb927c-ab23-47e3-b3eb-04e3e77e9d87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:24:44.722853 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.722792 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-kube-api-access-drxl8" (OuterVolumeSpecName: "kube-api-access-drxl8") pod "8adb927c-ab23-47e3-b3eb-04e3e77e9d87" (UID: "8adb927c-ab23-47e3-b3eb-04e3e77e9d87"). InnerVolumeSpecName "kube-api-access-drxl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:24:44.822300 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.822236 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-maas-api-tls\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:24:44.822300 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.822299 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-openshift-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:24:44.822514 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:44.822313 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drxl8\" (UniqueName: \"kubernetes.io/projected/8adb927c-ab23-47e3-b3eb-04e3e77e9d87-kube-api-access-drxl8\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:24:45.487835 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:45.487793 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" event={"ID":"8adb927c-ab23-47e3-b3eb-04e3e77e9d87","Type":"ContainerDied","Data":"a191a175eeab849814555af522a0781b519a261bc048366203fd7e53fcee8f22"} Jul 10 17:24:45.487835 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:45.487817 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq" Jul 10 17:24:45.488427 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:45.487860 2567 scope.go:117] "RemoveContainer" containerID="84f325f06583f4e6c1908dbb1d9eb2f6c6bd86d37edc20c729d75a056674482a" Jul 10 17:24:45.510776 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:45.510742 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq"] Jul 10 17:24:45.515643 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:45.515607 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9b2224f-5d77b66774-gbpdq"] Jul 10 17:24:46.682730 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:24:46.682694 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8adb927c-ab23-47e3-b3eb-04e3e77e9d87" path="/var/lib/kubelet/pods/8adb927c-ab23-47e3-b3eb-04e3e77e9d87/volumes" Jul 10 17:26:33.142371 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.142336 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd"] Jul 10 17:26:33.142820 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.142750 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8adb927c-ab23-47e3-b3eb-04e3e77e9d87" containerName="maas-api" Jul 10 17:26:33.142820 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.142762 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8adb927c-ab23-47e3-b3eb-04e3e77e9d87" containerName="maas-api" Jul 10 17:26:33.142956 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.142831 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8adb927c-ab23-47e3-b3eb-04e3e77e9d87" containerName="maas-api" Jul 10 17:26:33.145791 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.145762 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.148058 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.148035 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-cfd078-openshift-default-dockercfg-949lk\"" Jul 10 17:26:33.158851 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.158823 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd"] Jul 10 17:26:33.233822 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.233773 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-token\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.233822 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.233824 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-credential-socket\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.234100 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.233845 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-envoy\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.234100 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.233946 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-data\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.234100 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.233983 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-workload-certs\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.234100 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.234025 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istiod-ca-cert\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.234100 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.234057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng852\" (UniqueName: \"kubernetes.io/projected/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-kube-api-access-ng852\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.234100 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.234075 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-workload-socket\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.234771 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.234103 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-podinfo\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.335624 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.335580 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istiod-ca-cert\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.335624 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.335635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng852\" (UniqueName: \"kubernetes.io/projected/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-kube-api-access-ng852\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.335920 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.335662 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-workload-socket\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.335920 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.335810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-podinfo\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.335920 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.335882 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-token\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.336077 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.336046 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-credential-socket\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.336146 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.336088 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-workload-socket\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.336146 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.336098 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-envoy\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.336253 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.336203 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-data\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.336253 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.336239 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-workload-certs\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.336495 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.336469 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-credential-socket\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.336577 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.336494 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istiod-ca-cert\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.336821 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.336798 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-data\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.336949 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.336927 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-workload-certs\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.338349 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.338325 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-envoy\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.338507 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.338478 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-podinfo\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.343164 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.343134 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-token\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.343293 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.343222 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng852\" (UniqueName: \"kubernetes.io/projected/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-kube-api-access-ng852\") pod \"e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.456632 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.456596 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:33.603096 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.603067 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd"] Jul 10 17:26:33.605293 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:26:33.605240 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e95e78_7594_4701_a0fe_b9f9f93c0fb7.slice/crio-6e0891247971047e17250571a33ea72ce656ab0efdf5274a65ace0c272397416 WatchSource:0}: Error finding container 6e0891247971047e17250571a33ea72ce656ab0efdf5274a65ace0c272397416: Status 404 returned error can't find the container with id 6e0891247971047e17250571a33ea72ce656ab0efdf5274a65ace0c272397416 Jul 10 17:26:33.607359 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.607325 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:26:33.607427 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.607395 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:26:33.607427 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:33.607423 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:26:34.031659 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:34.031620 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" event={"ID":"42e95e78-7594-4701-a0fe-b9f9f93c0fb7","Type":"ContainerStarted","Data":"372e82bb2d0327dce8a884ea328b3804b05456d55afa4d6e6143262335eb1d5f"} Jul 10 17:26:34.031842 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:34.031675 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" event={"ID":"42e95e78-7594-4701-a0fe-b9f9f93c0fb7","Type":"ContainerStarted","Data":"6e0891247971047e17250571a33ea72ce656ab0efdf5274a65ace0c272397416"} Jul 10 17:26:34.052013 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:34.051959 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" podStartSLOduration=1.051941251 podStartE2EDuration="1.051941251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:26:34.05086868 +0000 UTC m=+1923.938160023" watchObservedRunningTime="2026-07-10 17:26:34.051941251 +0000 UTC m=+1923.939232571" Jul 10 17:26:34.457804 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:34.457765 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:34.462874 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:34.462848 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:35.037906 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:35.037879 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:35.038786 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:35.038765 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:36.229832 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.229800 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg"] Jul 10 17:26:36.233544 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.233518 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:36.235758 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.235735 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-cfd078-serving-cert\"" Jul 10 17:26:36.242550 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.242517 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg"] Jul 10 17:26:36.363715 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.363681 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85146738-0d7d-4880-a554-924f5b1caaa1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg\" (UID: \"85146738-0d7d-4880-a554-924f5b1caaa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:36.363887 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.363733 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85146738-0d7d-4880-a554-924f5b1caaa1-maas-api-tls\") pod \"maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg\" (UID: \"85146738-0d7d-4880-a554-924f5b1caaa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:36.363887 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.363796 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxr9n\" (UniqueName: \"kubernetes.io/projected/85146738-0d7d-4880-a554-924f5b1caaa1-kube-api-access-wxr9n\") pod \"maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg\" (UID: \"85146738-0d7d-4880-a554-924f5b1caaa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:36.464641 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.464603 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85146738-0d7d-4880-a554-924f5b1caaa1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg\" (UID: \"85146738-0d7d-4880-a554-924f5b1caaa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:36.464810 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.464653 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85146738-0d7d-4880-a554-924f5b1caaa1-maas-api-tls\") pod \"maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg\" (UID: \"85146738-0d7d-4880-a554-924f5b1caaa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:36.464810 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.464682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxr9n\" (UniqueName: \"kubernetes.io/projected/85146738-0d7d-4880-a554-924f5b1caaa1-kube-api-access-wxr9n\") pod \"maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg\" (UID: \"85146738-0d7d-4880-a554-924f5b1caaa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:36.464896 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:26:36.464824 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-serving-cert: secret "maas-api-e2e-inf-a-cfd078-serving-cert" not found Jul 10 17:26:36.464934 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:26:36.464904 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85146738-0d7d-4880-a554-924f5b1caaa1-maas-api-tls podName:85146738-0d7d-4880-a554-924f5b1caaa1 nodeName:}" failed. No retries permitted until 2026-07-10 17:26:36.964880859 +0000 UTC m=+1926.852172160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85146738-0d7d-4880-a554-924f5b1caaa1-maas-api-tls") pod "maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" (UID: "85146738-0d7d-4880-a554-924f5b1caaa1") : secret "maas-api-e2e-inf-a-cfd078-serving-cert" not found Jul 10 17:26:36.465345 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.465326 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85146738-0d7d-4880-a554-924f5b1caaa1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg\" (UID: \"85146738-0d7d-4880-a554-924f5b1caaa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:36.475675 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.475649 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxr9n\" (UniqueName: \"kubernetes.io/projected/85146738-0d7d-4880-a554-924f5b1caaa1-kube-api-access-wxr9n\") pod \"maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg\" (UID: \"85146738-0d7d-4880-a554-924f5b1caaa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:36.969375 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.969333 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85146738-0d7d-4880-a554-924f5b1caaa1-maas-api-tls\") pod \"maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg\" (UID: \"85146738-0d7d-4880-a554-924f5b1caaa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:36.972109 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:36.972083 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85146738-0d7d-4880-a554-924f5b1caaa1-maas-api-tls\") pod \"maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg\" (UID: \"85146738-0d7d-4880-a554-924f5b1caaa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:37.146686 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:37.146654 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:37.288038 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:37.287980 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg"] Jul 10 17:26:37.290332 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:26:37.290301 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85146738_0d7d_4880_a554_924f5b1caaa1.slice/crio-963a2109bfae98db44835355a0ffc02ae7282a38224f45986bf6b156fad9a746 WatchSource:0}: Error finding container 963a2109bfae98db44835355a0ffc02ae7282a38224f45986bf6b156fad9a746: Status 404 returned error can't find the container with id 963a2109bfae98db44835355a0ffc02ae7282a38224f45986bf6b156fad9a746 Jul 10 17:26:37.291853 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:37.291835 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:26:38.054755 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:38.054651 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" event={"ID":"85146738-0d7d-4880-a554-924f5b1caaa1","Type":"ContainerStarted","Data":"a51b1ff8a26edf88893b025ad3e370ddedf1a14c9d8c0a5947ed8497a138161a"} Jul 10 17:26:38.054755 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:38.054704 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" event={"ID":"85146738-0d7d-4880-a554-924f5b1caaa1","Type":"ContainerStarted","Data":"963a2109bfae98db44835355a0ffc02ae7282a38224f45986bf6b156fad9a746"} Jul 10 17:26:38.054987 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:38.054822 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:38.073375 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:38.073326 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" podStartSLOduration=1.587451135 podStartE2EDuration="2.073309265s" totalImagesPullingTime="485.85813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:26:38.071202469 +0000 UTC m=+1927.958493788" watchObservedRunningTime="2026-07-10 17:26:38.073309265 +0000 UTC m=+1927.960600584" Jul 10 17:26:44.067117 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:44.067086 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:26:44.842532 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:44.842495 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns"] Jul 10 17:26:44.846758 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:44.846726 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:44.849217 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:44.849189 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fc13d1-serving-cert\"" Jul 10 17:26:44.855376 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:44.855349 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns"] Jul 10 17:26:44.950217 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:44.950182 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm6dk\" (UniqueName: \"kubernetes.io/projected/46527a75-042e-4664-a7e4-fe16147f5d9c-kube-api-access-nm6dk\") pod \"maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns\" (UID: \"46527a75-042e-4664-a7e4-fe16147f5d9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:44.950433 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:44.950225 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46527a75-042e-4664-a7e4-fe16147f5d9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns\" (UID: \"46527a75-042e-4664-a7e4-fe16147f5d9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:44.950433 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:44.950247 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46527a75-042e-4664-a7e4-fe16147f5d9c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns\" (UID: \"46527a75-042e-4664-a7e4-fe16147f5d9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:45.051224 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:45.051187 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm6dk\" (UniqueName: \"kubernetes.io/projected/46527a75-042e-4664-a7e4-fe16147f5d9c-kube-api-access-nm6dk\") pod \"maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns\" (UID: \"46527a75-042e-4664-a7e4-fe16147f5d9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:45.051405 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:45.051232 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46527a75-042e-4664-a7e4-fe16147f5d9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns\" (UID: \"46527a75-042e-4664-a7e4-fe16147f5d9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:45.051405 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:45.051325 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46527a75-042e-4664-a7e4-fe16147f5d9c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns\" (UID: \"46527a75-042e-4664-a7e4-fe16147f5d9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:45.051405 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:26:45.051351 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-serving-cert: secret "maas-api-e2e-inf-b-fc13d1-serving-cert" not found Jul 10 17:26:45.051405 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:26:45.051414 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46527a75-042e-4664-a7e4-fe16147f5d9c-maas-api-tls podName:46527a75-042e-4664-a7e4-fe16147f5d9c nodeName:}" failed. No retries permitted until 2026-07-10 17:26:45.55139622 +0000 UTC m=+1935.438687518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46527a75-042e-4664-a7e4-fe16147f5d9c-maas-api-tls") pod "maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" (UID: "46527a75-042e-4664-a7e4-fe16147f5d9c") : secret "maas-api-e2e-inf-b-fc13d1-serving-cert" not found Jul 10 17:26:45.051949 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:45.051931 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46527a75-042e-4664-a7e4-fe16147f5d9c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns\" (UID: \"46527a75-042e-4664-a7e4-fe16147f5d9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:45.059965 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:45.059942 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm6dk\" (UniqueName: \"kubernetes.io/projected/46527a75-042e-4664-a7e4-fe16147f5d9c-kube-api-access-nm6dk\") pod \"maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns\" (UID: \"46527a75-042e-4664-a7e4-fe16147f5d9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:45.556099 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:45.556066 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46527a75-042e-4664-a7e4-fe16147f5d9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns\" (UID: \"46527a75-042e-4664-a7e4-fe16147f5d9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:45.558582 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:45.558561 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46527a75-042e-4664-a7e4-fe16147f5d9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns\" (UID: \"46527a75-042e-4664-a7e4-fe16147f5d9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:45.762999 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:45.762952 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:45.952503 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:45.952473 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns"] Jul 10 17:26:45.954500 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:26:45.954469 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46527a75_042e_4664_a7e4_fe16147f5d9c.slice/crio-a5c1f3156e496918310e6157bf6b7802e7601ec6f02ab201ec12af2cd78cef8d WatchSource:0}: Error finding container a5c1f3156e496918310e6157bf6b7802e7601ec6f02ab201ec12af2cd78cef8d: Status 404 returned error can't find the container with id a5c1f3156e496918310e6157bf6b7802e7601ec6f02ab201ec12af2cd78cef8d Jul 10 17:26:46.103926 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:46.103832 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" event={"ID":"46527a75-042e-4664-a7e4-fe16147f5d9c","Type":"ContainerStarted","Data":"a5c1f3156e496918310e6157bf6b7802e7601ec6f02ab201ec12af2cd78cef8d"} Jul 10 17:26:47.110859 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:47.110817 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" event={"ID":"46527a75-042e-4664-a7e4-fe16147f5d9c","Type":"ContainerStarted","Data":"23543320af2a74dbbee086dc5b1a8240a721af6e62c8dcfc1bc87f853242d0c8"} Jul 10 17:26:47.111263 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:47.110913 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:47.128872 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:47.128818 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" podStartSLOduration=2.490101877 podStartE2EDuration="3.128801725s" totalImagesPullingTime="638.699848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:26:47.126613893 +0000 UTC m=+1937.013905214" watchObservedRunningTime="2026-07-10 17:26:47.128801725 +0000 UTC m=+1937.016093304" Jul 10 17:26:51.079450 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:51.079406 2567 patch_prober.go:28] interesting pod/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.135.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 17:26:51.079946 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:51.079475 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" podUID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 17:26:53.122442 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:53.122410 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:26:53.146074 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:53.146037 2567 generic.go:356] "Generic (PLEG): container finished" podID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerID="372e82bb2d0327dce8a884ea328b3804b05456d55afa4d6e6143262335eb1d5f" exitCode=0 Jul 10 17:26:53.146255 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:53.146121 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" event={"ID":"42e95e78-7594-4701-a0fe-b9f9f93c0fb7","Type":"ContainerDied","Data":"372e82bb2d0327dce8a884ea328b3804b05456d55afa4d6e6143262335eb1d5f"} Jul 10 17:26:53.146590 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:53.146569 2567 scope.go:117] "RemoveContainer" containerID="372e82bb2d0327dce8a884ea328b3804b05456d55afa4d6e6143262335eb1d5f" Jul 10 17:26:53.147944 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:53.147912 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:26:53.148033 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:53.147997 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:26:53.148076 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:53.148052 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:26:54.153215 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:54.153180 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" event={"ID":"42e95e78-7594-4701-a0fe-b9f9f93c0fb7","Type":"ContainerStarted","Data":"4fd1e6abb01f85b0cceff09bf50883ff22623f9c39c5093b064c8a00140c3e56"} Jul 10 17:26:54.457197 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:54.457154 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:54.457486 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:54.457464 2567 patch_prober.go:28] interesting pod/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.64:15021/healthz/ready\": dial tcp 10.135.0.64:15021: connect: connection refused" start-of-body= Jul 10 17:26:54.457553 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:54.457527 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" podUID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.64:15021/healthz/ready\": dial tcp 10.135.0.64:15021: connect: connection refused" Jul 10 17:26:55.457554 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:55.457502 2567 patch_prober.go:28] interesting pod/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.64:15021/healthz/ready\": dial tcp 10.135.0.64:15021: connect: connection refused" start-of-body= Jul 10 17:26:55.458084 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:55.457586 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" podUID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.64:15021/healthz/ready\": dial tcp 10.135.0.64:15021: connect: connection refused" Jul 10 17:26:56.457844 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:56.457796 2567 patch_prober.go:28] interesting pod/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.64:15021/healthz/ready\": dial tcp 10.135.0.64:15021: connect: connection refused" start-of-body= Jul 10 17:26:56.458391 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:56.457864 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" podUID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.64:15021/healthz/ready\": dial tcp 10.135.0.64:15021: connect: connection refused" Jul 10 17:26:57.457391 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:57.457355 2567 patch_prober.go:28] interesting pod/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.64:15021/healthz/ready\": dial tcp 10.135.0.64:15021: connect: connection refused" start-of-body= Jul 10 17:26:57.457586 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:57.457413 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" podUID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.64:15021/healthz/ready\": dial tcp 10.135.0.64:15021: connect: connection refused" Jul 10 17:26:58.461005 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:58.460971 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:58.461424 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:58.461303 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:26:58.462083 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:26:58.462066 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:27:50.271052 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:50.271016 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg"] Jul 10 17:27:50.271640 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:50.271221 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" podUID="85146738-0d7d-4880-a554-924f5b1caaa1" containerName="maas-api" containerID="cri-o://a51b1ff8a26edf88893b025ad3e370ddedf1a14c9d8c0a5947ed8497a138161a" gracePeriod=30 Jul 10 17:27:50.718644 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:50.718619 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:27:50.909017 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:50.908925 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85146738-0d7d-4880-a554-924f5b1caaa1-openshift-service-ca\") pod \"85146738-0d7d-4880-a554-924f5b1caaa1\" (UID: \"85146738-0d7d-4880-a554-924f5b1caaa1\") " Jul 10 17:27:50.909017 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:50.908978 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85146738-0d7d-4880-a554-924f5b1caaa1-maas-api-tls\") pod \"85146738-0d7d-4880-a554-924f5b1caaa1\" (UID: \"85146738-0d7d-4880-a554-924f5b1caaa1\") " Jul 10 17:27:50.909251 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:50.909101 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxr9n\" (UniqueName: \"kubernetes.io/projected/85146738-0d7d-4880-a554-924f5b1caaa1-kube-api-access-wxr9n\") pod \"85146738-0d7d-4880-a554-924f5b1caaa1\" (UID: \"85146738-0d7d-4880-a554-924f5b1caaa1\") " Jul 10 17:27:50.909400 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:50.909378 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85146738-0d7d-4880-a554-924f5b1caaa1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85146738-0d7d-4880-a554-924f5b1caaa1" (UID: "85146738-0d7d-4880-a554-924f5b1caaa1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:27:50.911346 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:50.911314 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85146738-0d7d-4880-a554-924f5b1caaa1-kube-api-access-wxr9n" (OuterVolumeSpecName: "kube-api-access-wxr9n") pod "85146738-0d7d-4880-a554-924f5b1caaa1" (UID: "85146738-0d7d-4880-a554-924f5b1caaa1"). InnerVolumeSpecName "kube-api-access-wxr9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:27:50.911447 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:50.911382 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85146738-0d7d-4880-a554-924f5b1caaa1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85146738-0d7d-4880-a554-924f5b1caaa1" (UID: "85146738-0d7d-4880-a554-924f5b1caaa1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:27:51.010205 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:51.010169 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85146738-0d7d-4880-a554-924f5b1caaa1-openshift-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:27:51.010205 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:51.010200 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85146738-0d7d-4880-a554-924f5b1caaa1-maas-api-tls\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:27:51.010205 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:51.010209 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxr9n\" (UniqueName: \"kubernetes.io/projected/85146738-0d7d-4880-a554-924f5b1caaa1-kube-api-access-wxr9n\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:27:51.453908 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:51.453859 2567 generic.go:356] "Generic (PLEG): container finished" podID="85146738-0d7d-4880-a554-924f5b1caaa1" containerID="a51b1ff8a26edf88893b025ad3e370ddedf1a14c9d8c0a5947ed8497a138161a" exitCode=0 Jul 10 17:27:51.454414 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:51.453926 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" Jul 10 17:27:51.454414 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:51.453956 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" event={"ID":"85146738-0d7d-4880-a554-924f5b1caaa1","Type":"ContainerDied","Data":"a51b1ff8a26edf88893b025ad3e370ddedf1a14c9d8c0a5947ed8497a138161a"} Jul 10 17:27:51.454414 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:51.454038 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg" event={"ID":"85146738-0d7d-4880-a554-924f5b1caaa1","Type":"ContainerDied","Data":"963a2109bfae98db44835355a0ffc02ae7282a38224f45986bf6b156fad9a746"} Jul 10 17:27:51.454414 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:51.454060 2567 scope.go:117] "RemoveContainer" containerID="a51b1ff8a26edf88893b025ad3e370ddedf1a14c9d8c0a5947ed8497a138161a" Jul 10 17:27:51.463732 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:51.463712 2567 scope.go:117] "RemoveContainer" containerID="a51b1ff8a26edf88893b025ad3e370ddedf1a14c9d8c0a5947ed8497a138161a" Jul 10 17:27:51.464006 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:27:51.463988 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51b1ff8a26edf88893b025ad3e370ddedf1a14c9d8c0a5947ed8497a138161a\": container with ID starting with a51b1ff8a26edf88893b025ad3e370ddedf1a14c9d8c0a5947ed8497a138161a not found: ID does not exist" containerID="a51b1ff8a26edf88893b025ad3e370ddedf1a14c9d8c0a5947ed8497a138161a" Jul 10 17:27:51.464049 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:51.464017 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51b1ff8a26edf88893b025ad3e370ddedf1a14c9d8c0a5947ed8497a138161a"} err="failed to get container status \"a51b1ff8a26edf88893b025ad3e370ddedf1a14c9d8c0a5947ed8497a138161a\": rpc error: code = NotFound desc = could not find container \"a51b1ff8a26edf88893b025ad3e370ddedf1a14c9d8c0a5947ed8497a138161a\": container with ID starting with a51b1ff8a26edf88893b025ad3e370ddedf1a14c9d8c0a5947ed8497a138161a not found: ID does not exist" Jul 10 17:27:51.475550 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:51.475467 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg"] Jul 10 17:27:51.477428 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:51.477403 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cfd078-5dbf6864c7-fflbg"] Jul 10 17:27:52.682701 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:27:52.682669 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85146738-0d7d-4880-a554-924f5b1caaa1" path="/var/lib/kubelet/pods/85146738-0d7d-4880-a554-924f5b1caaa1/volumes" Jul 10 17:28:32.056653 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:32.056612 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd"] Jul 10 17:28:32.057147 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:32.056869 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" podUID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerName="istio-proxy" containerID="cri-o://4fd1e6abb01f85b0cceff09bf50883ff22623f9c39c5093b064c8a00140c3e56" gracePeriod=30 Jul 10 17:28:33.142055 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:33.142010 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns"] Jul 10 17:28:33.142560 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:33.142330 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" podUID="46527a75-042e-4664-a7e4-fe16147f5d9c" containerName="maas-api" containerID="cri-o://23543320af2a74dbbee086dc5b1a8240a721af6e62c8dcfc1bc87f853242d0c8" gracePeriod=30 Jul 10 17:28:34.407841 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.407809 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:28:34.424656 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.424616 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46527a75-042e-4664-a7e4-fe16147f5d9c-openshift-service-ca\") pod \"46527a75-042e-4664-a7e4-fe16147f5d9c\" (UID: \"46527a75-042e-4664-a7e4-fe16147f5d9c\") " Jul 10 17:28:34.424864 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.424776 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46527a75-042e-4664-a7e4-fe16147f5d9c-maas-api-tls\") pod \"46527a75-042e-4664-a7e4-fe16147f5d9c\" (UID: \"46527a75-042e-4664-a7e4-fe16147f5d9c\") " Jul 10 17:28:34.424864 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.424808 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm6dk\" (UniqueName: \"kubernetes.io/projected/46527a75-042e-4664-a7e4-fe16147f5d9c-kube-api-access-nm6dk\") pod \"46527a75-042e-4664-a7e4-fe16147f5d9c\" (UID: \"46527a75-042e-4664-a7e4-fe16147f5d9c\") " Jul 10 17:28:34.425379 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.425351 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46527a75-042e-4664-a7e4-fe16147f5d9c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46527a75-042e-4664-a7e4-fe16147f5d9c" (UID: "46527a75-042e-4664-a7e4-fe16147f5d9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:28:34.427422 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.427391 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46527a75-042e-4664-a7e4-fe16147f5d9c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46527a75-042e-4664-a7e4-fe16147f5d9c" (UID: "46527a75-042e-4664-a7e4-fe16147f5d9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:28:34.427565 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.427543 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46527a75-042e-4664-a7e4-fe16147f5d9c-kube-api-access-nm6dk" (OuterVolumeSpecName: "kube-api-access-nm6dk") pod "46527a75-042e-4664-a7e4-fe16147f5d9c" (UID: "46527a75-042e-4664-a7e4-fe16147f5d9c"). InnerVolumeSpecName "kube-api-access-nm6dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:28:34.526007 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.525963 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46527a75-042e-4664-a7e4-fe16147f5d9c-openshift-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:28:34.526007 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.525999 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46527a75-042e-4664-a7e4-fe16147f5d9c-maas-api-tls\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:28:34.526007 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.526008 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm6dk\" (UniqueName: \"kubernetes.io/projected/46527a75-042e-4664-a7e4-fe16147f5d9c-kube-api-access-nm6dk\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:28:34.673620 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.673580 2567 generic.go:356] "Generic (PLEG): container finished" podID="46527a75-042e-4664-a7e4-fe16147f5d9c" containerID="23543320af2a74dbbee086dc5b1a8240a721af6e62c8dcfc1bc87f853242d0c8" exitCode=0 Jul 10 17:28:34.673799 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.673644 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" Jul 10 17:28:34.673799 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.673662 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" event={"ID":"46527a75-042e-4664-a7e4-fe16147f5d9c","Type":"ContainerDied","Data":"23543320af2a74dbbee086dc5b1a8240a721af6e62c8dcfc1bc87f853242d0c8"} Jul 10 17:28:34.673799 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.673739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns" event={"ID":"46527a75-042e-4664-a7e4-fe16147f5d9c","Type":"ContainerDied","Data":"a5c1f3156e496918310e6157bf6b7802e7601ec6f02ab201ec12af2cd78cef8d"} Jul 10 17:28:34.673799 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.673769 2567 scope.go:117] "RemoveContainer" containerID="23543320af2a74dbbee086dc5b1a8240a721af6e62c8dcfc1bc87f853242d0c8" Jul 10 17:28:34.687385 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.687363 2567 scope.go:117] "RemoveContainer" containerID="23543320af2a74dbbee086dc5b1a8240a721af6e62c8dcfc1bc87f853242d0c8" Jul 10 17:28:34.687723 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:28:34.687697 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23543320af2a74dbbee086dc5b1a8240a721af6e62c8dcfc1bc87f853242d0c8\": container with ID starting with 23543320af2a74dbbee086dc5b1a8240a721af6e62c8dcfc1bc87f853242d0c8 not found: ID does not exist" containerID="23543320af2a74dbbee086dc5b1a8240a721af6e62c8dcfc1bc87f853242d0c8" Jul 10 17:28:34.687797 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.687737 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23543320af2a74dbbee086dc5b1a8240a721af6e62c8dcfc1bc87f853242d0c8"} err="failed to get container status \"23543320af2a74dbbee086dc5b1a8240a721af6e62c8dcfc1bc87f853242d0c8\": rpc error: code = NotFound desc = could not find container \"23543320af2a74dbbee086dc5b1a8240a721af6e62c8dcfc1bc87f853242d0c8\": container with ID starting with 23543320af2a74dbbee086dc5b1a8240a721af6e62c8dcfc1bc87f853242d0c8 not found: ID does not exist" Jul 10 17:28:34.699834 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.699803 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns"] Jul 10 17:28:34.703516 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:34.703488 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc13d1-d856cd46b-9qtns"] Jul 10 17:28:36.683973 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:36.683940 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46527a75-042e-4664-a7e4-fe16147f5d9c" path="/var/lib/kubelet/pods/46527a75-042e-4664-a7e4-fe16147f5d9c/volumes" Jul 10 17:28:37.317328 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.317253 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:28:37.355556 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.355525 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-envoy\") pod \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " Jul 10 17:28:37.355556 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.355561 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-workload-certs\") pod \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " Jul 10 17:28:37.355779 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.355605 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-workload-socket\") pod \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " Jul 10 17:28:37.355779 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.355663 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-credential-socket\") pod \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " Jul 10 17:28:37.355779 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.355710 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istiod-ca-cert\") pod \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " Jul 10 17:28:37.355779 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.355740 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng852\" (UniqueName: \"kubernetes.io/projected/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-kube-api-access-ng852\") pod \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " Jul 10 17:28:37.355779 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.355778 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-token\") pod \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " Jul 10 17:28:37.356037 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.355803 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-podinfo\") pod \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " Jul 10 17:28:37.356037 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.355838 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-data\") pod \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\" (UID: \"42e95e78-7594-4701-a0fe-b9f9f93c0fb7\") " Jul 10 17:28:37.356037 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.355874 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42e95e78-7594-4701-a0fe-b9f9f93c0fb7" (UID: "42e95e78-7594-4701-a0fe-b9f9f93c0fb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:37.356037 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.355979 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42e95e78-7594-4701-a0fe-b9f9f93c0fb7" (UID: "42e95e78-7594-4701-a0fe-b9f9f93c0fb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:37.356249 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.356175 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-workload-certs\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:28:37.356249 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.356196 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-workload-socket\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:28:37.356249 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.356218 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42e95e78-7594-4701-a0fe-b9f9f93c0fb7" (UID: "42e95e78-7594-4701-a0fe-b9f9f93c0fb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:37.356436 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.356322 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42e95e78-7594-4701-a0fe-b9f9f93c0fb7" (UID: "42e95e78-7594-4701-a0fe-b9f9f93c0fb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:28:37.357258 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.357229 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-data" (OuterVolumeSpecName: "istio-data") pod "42e95e78-7594-4701-a0fe-b9f9f93c0fb7" (UID: "42e95e78-7594-4701-a0fe-b9f9f93c0fb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:37.358110 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.358080 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-token" (OuterVolumeSpecName: "istio-token") pod "42e95e78-7594-4701-a0fe-b9f9f93c0fb7" (UID: "42e95e78-7594-4701-a0fe-b9f9f93c0fb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:28:37.358376 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.358343 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-kube-api-access-ng852" (OuterVolumeSpecName: "kube-api-access-ng852") pod "42e95e78-7594-4701-a0fe-b9f9f93c0fb7" (UID: "42e95e78-7594-4701-a0fe-b9f9f93c0fb7"). InnerVolumeSpecName "kube-api-access-ng852". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:28:37.358468 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.358378 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42e95e78-7594-4701-a0fe-b9f9f93c0fb7" (UID: "42e95e78-7594-4701-a0fe-b9f9f93c0fb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:37.358468 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.358397 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42e95e78-7594-4701-a0fe-b9f9f93c0fb7" (UID: "42e95e78-7594-4701-a0fe-b9f9f93c0fb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:28:37.456992 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.456954 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-token\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:28:37.456992 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.456984 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-podinfo\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:28:37.456992 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.456995 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-data\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:28:37.456992 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.457003 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istio-envoy\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:28:37.457266 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.457012 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-credential-socket\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:28:37.457266 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.457022 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-istiod-ca-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:28:37.457266 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.457030 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng852\" (UniqueName: \"kubernetes.io/projected/42e95e78-7594-4701-a0fe-b9f9f93c0fb7-kube-api-access-ng852\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:28:37.691825 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.691790 2567 generic.go:356] "Generic (PLEG): container finished" podID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerID="4fd1e6abb01f85b0cceff09bf50883ff22623f9c39c5093b064c8a00140c3e56" exitCode=0 Jul 10 17:28:37.692310 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.691852 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" Jul 10 17:28:37.692310 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.691852 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" event={"ID":"42e95e78-7594-4701-a0fe-b9f9f93c0fb7","Type":"ContainerDied","Data":"4fd1e6abb01f85b0cceff09bf50883ff22623f9c39c5093b064c8a00140c3e56"} Jul 10 17:28:37.692310 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.691928 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd" event={"ID":"42e95e78-7594-4701-a0fe-b9f9f93c0fb7","Type":"ContainerDied","Data":"6e0891247971047e17250571a33ea72ce656ab0efdf5274a65ace0c272397416"} Jul 10 17:28:37.692310 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.691961 2567 scope.go:117] "RemoveContainer" containerID="4fd1e6abb01f85b0cceff09bf50883ff22623f9c39c5093b064c8a00140c3e56" Jul 10 17:28:37.701633 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.701616 2567 scope.go:117] "RemoveContainer" containerID="372e82bb2d0327dce8a884ea328b3804b05456d55afa4d6e6143262335eb1d5f" Jul 10 17:28:37.709722 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.709703 2567 scope.go:117] "RemoveContainer" containerID="4fd1e6abb01f85b0cceff09bf50883ff22623f9c39c5093b064c8a00140c3e56" Jul 10 17:28:37.710052 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:28:37.710032 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd1e6abb01f85b0cceff09bf50883ff22623f9c39c5093b064c8a00140c3e56\": container with ID starting with 4fd1e6abb01f85b0cceff09bf50883ff22623f9c39c5093b064c8a00140c3e56 not found: ID does not exist" containerID="4fd1e6abb01f85b0cceff09bf50883ff22623f9c39c5093b064c8a00140c3e56" Jul 10 17:28:37.710104 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.710065 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd1e6abb01f85b0cceff09bf50883ff22623f9c39c5093b064c8a00140c3e56"} err="failed to get container status \"4fd1e6abb01f85b0cceff09bf50883ff22623f9c39c5093b064c8a00140c3e56\": rpc error: code = NotFound desc = could not find container \"4fd1e6abb01f85b0cceff09bf50883ff22623f9c39c5093b064c8a00140c3e56\": container with ID starting with 4fd1e6abb01f85b0cceff09bf50883ff22623f9c39c5093b064c8a00140c3e56 not found: ID does not exist" Jul 10 17:28:37.710104 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.710082 2567 scope.go:117] "RemoveContainer" containerID="372e82bb2d0327dce8a884ea328b3804b05456d55afa4d6e6143262335eb1d5f" Jul 10 17:28:37.710356 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:28:37.710338 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372e82bb2d0327dce8a884ea328b3804b05456d55afa4d6e6143262335eb1d5f\": container with ID starting with 372e82bb2d0327dce8a884ea328b3804b05456d55afa4d6e6143262335eb1d5f not found: ID does not exist" containerID="372e82bb2d0327dce8a884ea328b3804b05456d55afa4d6e6143262335eb1d5f" Jul 10 17:28:37.710408 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.710363 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372e82bb2d0327dce8a884ea328b3804b05456d55afa4d6e6143262335eb1d5f"} err="failed to get container status \"372e82bb2d0327dce8a884ea328b3804b05456d55afa4d6e6143262335eb1d5f\": rpc error: code = NotFound desc = could not find container \"372e82bb2d0327dce8a884ea328b3804b05456d55afa4d6e6143262335eb1d5f\": container with ID starting with 372e82bb2d0327dce8a884ea328b3804b05456d55afa4d6e6143262335eb1d5f not found: ID does not exist" Jul 10 17:28:37.713890 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.713864 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd"] Jul 10 17:28:37.717666 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:37.717645 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cfd078-openshift-default-7f458fcd69-2p2hd"] Jul 10 17:28:38.683177 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:38.683140 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" path="/var/lib/kubelet/pods/42e95e78-7594-4701-a0fe-b9f9f93c0fb7/volumes" Jul 10 17:28:39.986342 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:39.986308 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5"] Jul 10 17:28:39.986775 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:39.986703 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46527a75-042e-4664-a7e4-fe16147f5d9c" containerName="maas-api" Jul 10 17:28:39.986775 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:39.986720 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="46527a75-042e-4664-a7e4-fe16147f5d9c" containerName="maas-api" Jul 10 17:28:39.986775 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:39.986737 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerName="istio-proxy" Jul 10 17:28:39.986775 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:39.986746 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerName="istio-proxy" Jul 10 17:28:39.986775 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:39.986756 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85146738-0d7d-4880-a554-924f5b1caaa1" containerName="maas-api" Jul 10 17:28:39.986775 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:39.986766 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="85146738-0d7d-4880-a554-924f5b1caaa1" containerName="maas-api" Jul 10 17:28:39.986993 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:39.986863 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerName="istio-proxy" Jul 10 17:28:39.986993 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:39.986873 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="46527a75-042e-4664-a7e4-fe16147f5d9c" containerName="maas-api" Jul 10 17:28:39.986993 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:39.986881 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="85146738-0d7d-4880-a554-924f5b1caaa1" containerName="maas-api" Jul 10 17:28:39.986993 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:39.986889 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerName="istio-proxy" Jul 10 17:28:39.991600 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:39.991573 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:39.994045 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:39.994017 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1775b6-openshift-default-dockercfg-zggqr\"" Jul 10 17:28:40.001405 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.001378 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5"] Jul 10 17:28:40.084452 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.084421 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-workload-certs\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.084452 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.084455 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-podinfo\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.084772 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.084479 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-data\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.084772 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.084519 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-token\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.084772 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.084658 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-workload-socket\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.084772 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.084692 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-envoy\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.084980 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.084836 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj88c\" (UniqueName: \"kubernetes.io/projected/3432fd86-c319-4f26-b5c8-d9a2920386dc-kube-api-access-mj88c\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.084980 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.084880 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-credential-socket\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.084980 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.084904 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3432fd86-c319-4f26-b5c8-d9a2920386dc-istiod-ca-cert\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186009 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.185966 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-workload-certs\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186009 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.186007 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-podinfo\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186319 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.186031 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-data\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186319 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.186051 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-token\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186319 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.186084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-workload-socket\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186319 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.186105 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-envoy\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186319 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.186232 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj88c\" (UniqueName: \"kubernetes.io/projected/3432fd86-c319-4f26-b5c8-d9a2920386dc-kube-api-access-mj88c\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186319 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.186305 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-credential-socket\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186627 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.186330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3432fd86-c319-4f26-b5c8-d9a2920386dc-istiod-ca-cert\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186627 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.186522 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-data\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186627 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.186529 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-workload-socket\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186741 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.186711 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-credential-socket\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186826 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.186797 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-workload-certs\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.186970 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.186951 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3432fd86-c319-4f26-b5c8-d9a2920386dc-istiod-ca-cert\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.188616 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.188593 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-envoy\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.188912 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.188894 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-podinfo\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.193662 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.193631 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-token\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.193862 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.193845 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj88c\" (UniqueName: \"kubernetes.io/projected/3432fd86-c319-4f26-b5c8-d9a2920386dc-kube-api-access-mj88c\") pod \"e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.303437 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.303349 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:40.450364 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.450339 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5"] Jul 10 17:28:40.450945 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:28:40.450917 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3432fd86_c319_4f26_b5c8_d9a2920386dc.slice/crio-3ceceb42fd562d51f6d05272f602e0ae96e7f308f272a5c2648fa38bca86b135 WatchSource:0}: Error finding container 3ceceb42fd562d51f6d05272f602e0ae96e7f308f272a5c2648fa38bca86b135: Status 404 returned error can't find the container with id 3ceceb42fd562d51f6d05272f602e0ae96e7f308f272a5c2648fa38bca86b135 Jul 10 17:28:40.453385 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.453347 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:28:40.453449 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.453432 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:28:40.453503 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.453487 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:28:40.711344 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.711306 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" event={"ID":"3432fd86-c319-4f26-b5c8-d9a2920386dc","Type":"ContainerStarted","Data":"aa8a0a808f9c9837822db1176e33102621abb9590ab500938917ed0abae53c8c"} Jul 10 17:28:40.711554 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:40.711368 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" event={"ID":"3432fd86-c319-4f26-b5c8-d9a2920386dc","Type":"ContainerStarted","Data":"3ceceb42fd562d51f6d05272f602e0ae96e7f308f272a5c2648fa38bca86b135"} Jul 10 17:28:41.303916 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:41.303874 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:41.308841 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:41.308815 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:41.329324 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:41.329263 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" podStartSLOduration=2.329249962 podStartE2EDuration="2.329249962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:28:40.736053333 +0000 UTC m=+2050.623344656" watchObservedRunningTime="2026-07-10 17:28:41.329249962 +0000 UTC m=+2051.216541282" Jul 10 17:28:41.717846 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:41.717804 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:41.718622 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:41.718599 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:28:42.944115 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:42.944080 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5"] Jul 10 17:28:42.944531 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:42.944481 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerName="istio-proxy" Jul 10 17:28:42.944531 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:42.944495 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e95e78-7594-4701-a0fe-b9f9f93c0fb7" containerName="istio-proxy" Jul 10 17:28:42.947837 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:42.947821 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:42.950058 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:42.950035 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1775b6-serving-cert\"" Jul 10 17:28:42.957212 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:42.957184 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5"] Jul 10 17:28:43.012925 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:43.012896 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkqql\" (UniqueName: \"kubernetes.io/projected/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-kube-api-access-jkqql\") pod \"maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5\" (UID: \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:43.013097 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:43.012934 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-openshift-service-ca\") pod \"maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5\" (UID: \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:43.013097 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:43.012977 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-maas-api-tls\") pod \"maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5\" (UID: \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:43.114491 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:43.114452 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkqql\" (UniqueName: \"kubernetes.io/projected/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-kube-api-access-jkqql\") pod \"maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5\" (UID: \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:43.114491 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:43.114497 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-openshift-service-ca\") pod \"maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5\" (UID: \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:43.114748 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:43.114542 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-maas-api-tls\") pod \"maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5\" (UID: \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:43.114748 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:28:43.114656 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-serving-cert: secret "maas-api-e2e-api-a-1775b6-serving-cert" not found Jul 10 17:28:43.114748 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:28:43.114710 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-maas-api-tls podName:856eee9d-546f-42cc-9f3d-d0f46bd1ac14 nodeName:}" failed. No retries permitted until 2026-07-10 17:28:43.614692064 +0000 UTC m=+2053.501983362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-maas-api-tls") pod "maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" (UID: "856eee9d-546f-42cc-9f3d-d0f46bd1ac14") : secret "maas-api-e2e-api-a-1775b6-serving-cert" not found Jul 10 17:28:43.115124 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:43.115103 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-openshift-service-ca\") pod \"maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5\" (UID: \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:43.127169 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:43.127141 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkqql\" (UniqueName: \"kubernetes.io/projected/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-kube-api-access-jkqql\") pod \"maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5\" (UID: \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:43.620512 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:43.620464 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-maas-api-tls\") pod \"maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5\" (UID: \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:43.623158 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:43.623124 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-maas-api-tls\") pod \"maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5\" (UID: \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:43.861414 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:43.861379 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:44.000771 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:44.000741 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5"] Jul 10 17:28:44.005068 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:28:44.005034 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856eee9d_546f_42cc_9f3d_d0f46bd1ac14.slice/crio-1b0e85b069bd8ae011b6d25acbd2777a34344bf9a2a7dea8935d0f9a3b862c13 WatchSource:0}: Error finding container 1b0e85b069bd8ae011b6d25acbd2777a34344bf9a2a7dea8935d0f9a3b862c13: Status 404 returned error can't find the container with id 1b0e85b069bd8ae011b6d25acbd2777a34344bf9a2a7dea8935d0f9a3b862c13 Jul 10 17:28:44.735649 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:44.735614 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" event={"ID":"856eee9d-546f-42cc-9f3d-d0f46bd1ac14","Type":"ContainerStarted","Data":"819e929bee1c02394cdbff761ea3a4c0a849c7c9e4d86a192f492165edcbd258"} Jul 10 17:28:44.735824 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:44.735676 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" event={"ID":"856eee9d-546f-42cc-9f3d-d0f46bd1ac14","Type":"ContainerStarted","Data":"1b0e85b069bd8ae011b6d25acbd2777a34344bf9a2a7dea8935d0f9a3b862c13"} Jul 10 17:28:44.735824 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:44.735706 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:44.753585 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:44.753535 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" podStartSLOduration=2.234321153 podStartE2EDuration="2.753520159s" totalImagesPullingTime="519.199006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:28:44.751914368 +0000 UTC m=+2054.639205690" watchObservedRunningTime="2026-07-10 17:28:44.753520159 +0000 UTC m=+2054.640811492" Jul 10 17:28:50.748006 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:50.747905 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:28:51.575926 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:51.575886 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7"] Jul 10 17:28:51.580170 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:51.580144 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:28:51.582657 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:51.582635 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ad6ba3-serving-cert\"" Jul 10 17:28:51.589197 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:51.589167 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7"] Jul 10 17:28:51.696629 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:51.696589 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6937ddba-5199-4c24-8f94-b47ae161f7ba-maas-api-tls\") pod \"maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7\" (UID: \"6937ddba-5199-4c24-8f94-b47ae161f7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:28:51.696908 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:51.696883 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfwpq\" (UniqueName: \"kubernetes.io/projected/6937ddba-5199-4c24-8f94-b47ae161f7ba-kube-api-access-qfwpq\") pod \"maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7\" (UID: \"6937ddba-5199-4c24-8f94-b47ae161f7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:28:51.697092 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:51.697048 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6937ddba-5199-4c24-8f94-b47ae161f7ba-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7\" (UID: \"6937ddba-5199-4c24-8f94-b47ae161f7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:28:51.798042 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:51.798004 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6937ddba-5199-4c24-8f94-b47ae161f7ba-maas-api-tls\") pod \"maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7\" (UID: \"6937ddba-5199-4c24-8f94-b47ae161f7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:28:51.798529 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:51.798051 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfwpq\" (UniqueName: \"kubernetes.io/projected/6937ddba-5199-4c24-8f94-b47ae161f7ba-kube-api-access-qfwpq\") pod \"maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7\" (UID: \"6937ddba-5199-4c24-8f94-b47ae161f7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:28:51.798529 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:51.798095 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6937ddba-5199-4c24-8f94-b47ae161f7ba-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7\" (UID: \"6937ddba-5199-4c24-8f94-b47ae161f7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:28:51.798529 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:28:51.798165 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-serving-cert: secret "maas-api-e2e-api-b-ad6ba3-serving-cert" not found Jul 10 17:28:51.798529 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:28:51.798234 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6937ddba-5199-4c24-8f94-b47ae161f7ba-maas-api-tls podName:6937ddba-5199-4c24-8f94-b47ae161f7ba nodeName:}" failed. No retries permitted until 2026-07-10 17:28:52.298215614 +0000 UTC m=+2062.185506912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6937ddba-5199-4c24-8f94-b47ae161f7ba-maas-api-tls") pod "maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" (UID: "6937ddba-5199-4c24-8f94-b47ae161f7ba") : secret "maas-api-e2e-api-b-ad6ba3-serving-cert" not found Jul 10 17:28:51.798922 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:51.798897 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6937ddba-5199-4c24-8f94-b47ae161f7ba-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7\" (UID: \"6937ddba-5199-4c24-8f94-b47ae161f7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:28:51.811804 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:51.811777 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfwpq\" (UniqueName: \"kubernetes.io/projected/6937ddba-5199-4c24-8f94-b47ae161f7ba-kube-api-access-qfwpq\") pod \"maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7\" (UID: \"6937ddba-5199-4c24-8f94-b47ae161f7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:28:52.303710 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:52.303657 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6937ddba-5199-4c24-8f94-b47ae161f7ba-maas-api-tls\") pod \"maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7\" (UID: \"6937ddba-5199-4c24-8f94-b47ae161f7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:28:52.306456 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:52.306431 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6937ddba-5199-4c24-8f94-b47ae161f7ba-maas-api-tls\") pod \"maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7\" (UID: \"6937ddba-5199-4c24-8f94-b47ae161f7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:28:52.493133 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:52.493092 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:28:52.667676 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:52.664804 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7"] Jul 10 17:28:52.669164 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:28:52.669133 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6937ddba_5199_4c24_8f94_b47ae161f7ba.slice/crio-543fa6128e475432399de57f10cacb737e5de8071fb9897b255d2b7bcbc18bd6 WatchSource:0}: Error finding container 543fa6128e475432399de57f10cacb737e5de8071fb9897b255d2b7bcbc18bd6: Status 404 returned error can't find the container with id 543fa6128e475432399de57f10cacb737e5de8071fb9897b255d2b7bcbc18bd6 Jul 10 17:28:52.777888 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:52.777827 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" event={"ID":"6937ddba-5199-4c24-8f94-b47ae161f7ba","Type":"ContainerStarted","Data":"543fa6128e475432399de57f10cacb737e5de8071fb9897b255d2b7bcbc18bd6"} Jul 10 17:28:53.787043 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:53.786995 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" event={"ID":"6937ddba-5199-4c24-8f94-b47ae161f7ba","Type":"ContainerStarted","Data":"9e1989de82a2be30412ed188380e703f397186cbefbebd972abc6b5977e90f45"} Jul 10 17:28:53.787928 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:53.787901 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:28:53.806685 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:53.806626 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" podStartSLOduration=2.395322978 podStartE2EDuration="2.806610563s" totalImagesPullingTime="411.287585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:28:53.805191026 +0000 UTC m=+2063.692482350" watchObservedRunningTime="2026-07-10 17:28:53.806610563 +0000 UTC m=+2063.693901923" Jul 10 17:28:57.761442 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:57.761390 2567 patch_prober.go:28] interesting pod/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.135.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 17:28:57.761904 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:57.761464 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" podUID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 17:28:59.830091 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:59.830051 2567 generic.go:356] "Generic (PLEG): container finished" podID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerID="aa8a0a808f9c9837822db1176e33102621abb9590ab500938917ed0abae53c8c" exitCode=0 Jul 10 17:28:59.830561 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:59.830120 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" event={"ID":"3432fd86-c319-4f26-b5c8-d9a2920386dc","Type":"ContainerDied","Data":"aa8a0a808f9c9837822db1176e33102621abb9590ab500938917ed0abae53c8c"} Jul 10 17:28:59.830561 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:59.830550 2567 scope.go:117] "RemoveContainer" containerID="aa8a0a808f9c9837822db1176e33102621abb9590ab500938917ed0abae53c8c" Jul 10 17:28:59.831691 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:59.831660 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:28:59.831794 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:59.831749 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:28:59.831855 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:28:59.831798 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:29:00.812804 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:00.812771 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:29:00.838373 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:00.838330 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" event={"ID":"3432fd86-c319-4f26-b5c8-d9a2920386dc","Type":"ContainerStarted","Data":"5bda053531e2ef461d2df6060cd5c208c3e9dc9640d854726f28a5b4161ae823"} Jul 10 17:29:01.304004 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:01.303966 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:29:01.304318 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:01.304298 2567 patch_prober.go:28] interesting pod/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.67:15021/healthz/ready\": dial tcp 10.135.0.67:15021: connect: connection refused" start-of-body= Jul 10 17:29:01.304376 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:01.304347 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" podUID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.67:15021/healthz/ready\": dial tcp 10.135.0.67:15021: connect: connection refused" Jul 10 17:29:02.304833 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:02.304781 2567 patch_prober.go:28] interesting pod/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.67:15021/healthz/ready\": dial tcp 10.135.0.67:15021: connect: connection refused" start-of-body= Jul 10 17:29:02.305369 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:02.304872 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" podUID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.67:15021/healthz/ready\": dial tcp 10.135.0.67:15021: connect: connection refused" Jul 10 17:29:03.304328 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:03.304264 2567 patch_prober.go:28] interesting pod/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.67:15021/healthz/ready\": dial tcp 10.135.0.67:15021: connect: connection refused" start-of-body= Jul 10 17:29:03.304519 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:03.304364 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" podUID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.67:15021/healthz/ready\": dial tcp 10.135.0.67:15021: connect: connection refused" Jul 10 17:29:04.304622 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:04.304582 2567 patch_prober.go:28] interesting pod/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.67:15021/healthz/ready\": dial tcp 10.135.0.67:15021: connect: connection refused" start-of-body= Jul 10 17:29:04.305013 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:04.304647 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" podUID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.67:15021/healthz/ready\": dial tcp 10.135.0.67:15021: connect: connection refused" Jul 10 17:29:04.634736 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:04.634660 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5"] Jul 10 17:29:04.634928 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:04.634903 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" podUID="856eee9d-546f-42cc-9f3d-d0f46bd1ac14" containerName="maas-api" containerID="cri-o://819e929bee1c02394cdbff761ea3a4c0a849c7c9e4d86a192f492165edcbd258" gracePeriod=30 Jul 10 17:29:04.859798 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:04.859766 2567 generic.go:356] "Generic (PLEG): container finished" podID="856eee9d-546f-42cc-9f3d-d0f46bd1ac14" containerID="819e929bee1c02394cdbff761ea3a4c0a849c7c9e4d86a192f492165edcbd258" exitCode=0 Jul 10 17:29:04.859968 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:04.859852 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" event={"ID":"856eee9d-546f-42cc-9f3d-d0f46bd1ac14","Type":"ContainerDied","Data":"819e929bee1c02394cdbff761ea3a4c0a849c7c9e4d86a192f492165edcbd258"} Jul 10 17:29:04.898624 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:04.898555 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:29:05.041097 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.041064 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-openshift-service-ca\") pod \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\" (UID: \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\") " Jul 10 17:29:05.041308 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.041119 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-maas-api-tls\") pod \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\" (UID: \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\") " Jul 10 17:29:05.041308 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.041247 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkqql\" (UniqueName: \"kubernetes.io/projected/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-kube-api-access-jkqql\") pod \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\" (UID: \"856eee9d-546f-42cc-9f3d-d0f46bd1ac14\") " Jul 10 17:29:05.041512 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.041491 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "856eee9d-546f-42cc-9f3d-d0f46bd1ac14" (UID: "856eee9d-546f-42cc-9f3d-d0f46bd1ac14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:29:05.041626 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.041611 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-openshift-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:05.043498 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.043476 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-kube-api-access-jkqql" (OuterVolumeSpecName: "kube-api-access-jkqql") pod "856eee9d-546f-42cc-9f3d-d0f46bd1ac14" (UID: "856eee9d-546f-42cc-9f3d-d0f46bd1ac14"). InnerVolumeSpecName "kube-api-access-jkqql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:29:05.043498 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.043483 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "856eee9d-546f-42cc-9f3d-d0f46bd1ac14" (UID: "856eee9d-546f-42cc-9f3d-d0f46bd1ac14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:29:05.142108 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.142068 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkqql\" (UniqueName: \"kubernetes.io/projected/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-kube-api-access-jkqql\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:05.142108 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.142101 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856eee9d-546f-42cc-9f3d-d0f46bd1ac14-maas-api-tls\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:05.307644 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.307611 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:29:05.308023 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.307884 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:29:05.308509 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.308491 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:29:05.865979 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.865949 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" Jul 10 17:29:05.866198 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.865990 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5" event={"ID":"856eee9d-546f-42cc-9f3d-d0f46bd1ac14","Type":"ContainerDied","Data":"1b0e85b069bd8ae011b6d25acbd2777a34344bf9a2a7dea8935d0f9a3b862c13"} Jul 10 17:29:05.866198 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.866052 2567 scope.go:117] "RemoveContainer" containerID="819e929bee1c02394cdbff761ea3a4c0a849c7c9e4d86a192f492165edcbd258" Jul 10 17:29:05.889332 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.889301 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5"] Jul 10 17:29:05.893858 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:05.893830 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1775b6-5ddc7895d4-d8jp5"] Jul 10 17:29:06.689748 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:06.689685 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="856eee9d-546f-42cc-9f3d-d0f46bd1ac14" path="/var/lib/kubelet/pods/856eee9d-546f-42cc-9f3d-d0f46bd1ac14/volumes" Jul 10 17:29:11.309161 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:11.309128 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5"] Jul 10 17:29:11.309631 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:11.309359 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" podUID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerName="istio-proxy" containerID="cri-o://5bda053531e2ef461d2df6060cd5c208c3e9dc9640d854726f28a5b4161ae823" gracePeriod=30 Jul 10 17:29:12.650024 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:12.649993 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7"] Jul 10 17:29:12.650478 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:12.650212 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" podUID="6937ddba-5199-4c24-8f94-b47ae161f7ba" containerName="maas-api" containerID="cri-o://9e1989de82a2be30412ed188380e703f397186cbefbebd972abc6b5977e90f45" gracePeriod=30 Jul 10 17:29:12.909710 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:12.909639 2567 generic.go:356] "Generic (PLEG): container finished" podID="6937ddba-5199-4c24-8f94-b47ae161f7ba" containerID="9e1989de82a2be30412ed188380e703f397186cbefbebd972abc6b5977e90f45" exitCode=0 Jul 10 17:29:12.909710 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:12.909671 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" event={"ID":"6937ddba-5199-4c24-8f94-b47ae161f7ba","Type":"ContainerDied","Data":"9e1989de82a2be30412ed188380e703f397186cbefbebd972abc6b5977e90f45"} Jul 10 17:29:12.909887 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:12.909758 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" event={"ID":"6937ddba-5199-4c24-8f94-b47ae161f7ba","Type":"ContainerDied","Data":"543fa6128e475432399de57f10cacb737e5de8071fb9897b255d2b7bcbc18bd6"} Jul 10 17:29:12.909887 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:12.909781 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="543fa6128e475432399de57f10cacb737e5de8071fb9897b255d2b7bcbc18bd6" Jul 10 17:29:12.912003 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:12.911986 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:29:13.015506 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:13.015473 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6937ddba-5199-4c24-8f94-b47ae161f7ba-maas-api-tls\") pod \"6937ddba-5199-4c24-8f94-b47ae161f7ba\" (UID: \"6937ddba-5199-4c24-8f94-b47ae161f7ba\") " Jul 10 17:29:13.015665 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:13.015520 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfwpq\" (UniqueName: \"kubernetes.io/projected/6937ddba-5199-4c24-8f94-b47ae161f7ba-kube-api-access-qfwpq\") pod \"6937ddba-5199-4c24-8f94-b47ae161f7ba\" (UID: \"6937ddba-5199-4c24-8f94-b47ae161f7ba\") " Jul 10 17:29:13.015665 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:13.015561 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6937ddba-5199-4c24-8f94-b47ae161f7ba-openshift-service-ca\") pod \"6937ddba-5199-4c24-8f94-b47ae161f7ba\" (UID: \"6937ddba-5199-4c24-8f94-b47ae161f7ba\") " Jul 10 17:29:13.015975 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:13.015952 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6937ddba-5199-4c24-8f94-b47ae161f7ba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6937ddba-5199-4c24-8f94-b47ae161f7ba" (UID: "6937ddba-5199-4c24-8f94-b47ae161f7ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:29:13.017667 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:13.017644 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6937ddba-5199-4c24-8f94-b47ae161f7ba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6937ddba-5199-4c24-8f94-b47ae161f7ba" (UID: "6937ddba-5199-4c24-8f94-b47ae161f7ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:29:13.017792 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:13.017769 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6937ddba-5199-4c24-8f94-b47ae161f7ba-kube-api-access-qfwpq" (OuterVolumeSpecName: "kube-api-access-qfwpq") pod "6937ddba-5199-4c24-8f94-b47ae161f7ba" (UID: "6937ddba-5199-4c24-8f94-b47ae161f7ba"). InnerVolumeSpecName "kube-api-access-qfwpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:29:13.116873 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:13.116843 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6937ddba-5199-4c24-8f94-b47ae161f7ba-openshift-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:13.116873 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:13.116872 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6937ddba-5199-4c24-8f94-b47ae161f7ba-maas-api-tls\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:13.117048 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:13.116881 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfwpq\" (UniqueName: \"kubernetes.io/projected/6937ddba-5199-4c24-8f94-b47ae161f7ba-kube-api-access-qfwpq\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:13.914713 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:13.914682 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7" Jul 10 17:29:13.935348 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:13.935313 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7"] Jul 10 17:29:13.937237 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:13.937215 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad6ba3-566c6ccd86-dl2v7"] Jul 10 17:29:14.684082 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:14.684040 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6937ddba-5199-4c24-8f94-b47ae161f7ba" path="/var/lib/kubelet/pods/6937ddba-5199-4c24-8f94-b47ae161f7ba/volumes" Jul 10 17:29:16.564383 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.564354 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:29:16.648324 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648211 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-token\") pod \"3432fd86-c319-4f26-b5c8-d9a2920386dc\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " Jul 10 17:29:16.648324 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648256 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-workload-socket\") pod \"3432fd86-c319-4f26-b5c8-d9a2920386dc\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " Jul 10 17:29:16.648324 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648303 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj88c\" (UniqueName: \"kubernetes.io/projected/3432fd86-c319-4f26-b5c8-d9a2920386dc-kube-api-access-mj88c\") pod \"3432fd86-c319-4f26-b5c8-d9a2920386dc\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " Jul 10 17:29:16.648629 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648361 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-credential-socket\") pod \"3432fd86-c319-4f26-b5c8-d9a2920386dc\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " Jul 10 17:29:16.648629 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648390 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-data\") pod \"3432fd86-c319-4f26-b5c8-d9a2920386dc\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " Jul 10 17:29:16.648629 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648427 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-workload-certs\") pod \"3432fd86-c319-4f26-b5c8-d9a2920386dc\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " Jul 10 17:29:16.648629 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648444 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3432fd86-c319-4f26-b5c8-d9a2920386dc-istiod-ca-cert\") pod \"3432fd86-c319-4f26-b5c8-d9a2920386dc\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " Jul 10 17:29:16.648629 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648479 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-envoy\") pod \"3432fd86-c319-4f26-b5c8-d9a2920386dc\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " Jul 10 17:29:16.648629 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648537 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-podinfo\") pod \"3432fd86-c319-4f26-b5c8-d9a2920386dc\" (UID: \"3432fd86-c319-4f26-b5c8-d9a2920386dc\") " Jul 10 17:29:16.648915 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648692 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3432fd86-c319-4f26-b5c8-d9a2920386dc" (UID: "3432fd86-c319-4f26-b5c8-d9a2920386dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:29:16.648915 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648742 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3432fd86-c319-4f26-b5c8-d9a2920386dc" (UID: "3432fd86-c319-4f26-b5c8-d9a2920386dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:29:16.648915 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648865 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-workload-socket\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:16.648915 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648884 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-workload-certs\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:16.649086 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648932 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3432fd86-c319-4f26-b5c8-d9a2920386dc" (UID: "3432fd86-c319-4f26-b5c8-d9a2920386dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:29:16.649086 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.648867 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3432fd86-c319-4f26-b5c8-d9a2920386dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3432fd86-c319-4f26-b5c8-d9a2920386dc" (UID: "3432fd86-c319-4f26-b5c8-d9a2920386dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:29:16.649906 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.649882 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "3432fd86-c319-4f26-b5c8-d9a2920386dc" (UID: "3432fd86-c319-4f26-b5c8-d9a2920386dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:29:16.650927 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.650897 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3432fd86-c319-4f26-b5c8-d9a2920386dc" (UID: "3432fd86-c319-4f26-b5c8-d9a2920386dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:29:16.651029 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.650975 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "3432fd86-c319-4f26-b5c8-d9a2920386dc" (UID: "3432fd86-c319-4f26-b5c8-d9a2920386dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:29:16.651209 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.651192 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3432fd86-c319-4f26-b5c8-d9a2920386dc-kube-api-access-mj88c" (OuterVolumeSpecName: "kube-api-access-mj88c") pod "3432fd86-c319-4f26-b5c8-d9a2920386dc" (UID: "3432fd86-c319-4f26-b5c8-d9a2920386dc"). InnerVolumeSpecName "kube-api-access-mj88c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:29:16.651252 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.651213 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3432fd86-c319-4f26-b5c8-d9a2920386dc" (UID: "3432fd86-c319-4f26-b5c8-d9a2920386dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:29:16.749988 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.749948 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-data\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:16.749988 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.749988 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3432fd86-c319-4f26-b5c8-d9a2920386dc-istiod-ca-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:16.750228 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.750001 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-envoy\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:16.750228 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.750009 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-podinfo\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:16.750228 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.750020 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3432fd86-c319-4f26-b5c8-d9a2920386dc-istio-token\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:16.750228 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.750028 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj88c\" (UniqueName: \"kubernetes.io/projected/3432fd86-c319-4f26-b5c8-d9a2920386dc-kube-api-access-mj88c\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:16.750228 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.750037 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3432fd86-c319-4f26-b5c8-d9a2920386dc-credential-socket\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:29:16.931462 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.931367 2567 generic.go:356] "Generic (PLEG): container finished" podID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerID="5bda053531e2ef461d2df6060cd5c208c3e9dc9640d854726f28a5b4161ae823" exitCode=0 Jul 10 17:29:16.931462 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.931417 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" Jul 10 17:29:16.931462 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.931456 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" event={"ID":"3432fd86-c319-4f26-b5c8-d9a2920386dc","Type":"ContainerDied","Data":"5bda053531e2ef461d2df6060cd5c208c3e9dc9640d854726f28a5b4161ae823"} Jul 10 17:29:16.931738 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.931513 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5" event={"ID":"3432fd86-c319-4f26-b5c8-d9a2920386dc","Type":"ContainerDied","Data":"3ceceb42fd562d51f6d05272f602e0ae96e7f308f272a5c2648fa38bca86b135"} Jul 10 17:29:16.931738 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.931536 2567 scope.go:117] "RemoveContainer" containerID="5bda053531e2ef461d2df6060cd5c208c3e9dc9640d854726f28a5b4161ae823" Jul 10 17:29:16.941334 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.941312 2567 scope.go:117] "RemoveContainer" containerID="aa8a0a808f9c9837822db1176e33102621abb9590ab500938917ed0abae53c8c" Jul 10 17:29:16.947359 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.947328 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5"] Jul 10 17:29:16.950446 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.950420 2567 scope.go:117] "RemoveContainer" containerID="5bda053531e2ef461d2df6060cd5c208c3e9dc9640d854726f28a5b4161ae823" Jul 10 17:29:16.950865 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:29:16.950828 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bda053531e2ef461d2df6060cd5c208c3e9dc9640d854726f28a5b4161ae823\": container with ID starting with 5bda053531e2ef461d2df6060cd5c208c3e9dc9640d854726f28a5b4161ae823 not found: ID does not exist" containerID="5bda053531e2ef461d2df6060cd5c208c3e9dc9640d854726f28a5b4161ae823" Jul 10 17:29:16.950983 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.950877 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bda053531e2ef461d2df6060cd5c208c3e9dc9640d854726f28a5b4161ae823"} err="failed to get container status \"5bda053531e2ef461d2df6060cd5c208c3e9dc9640d854726f28a5b4161ae823\": rpc error: code = NotFound desc = could not find container \"5bda053531e2ef461d2df6060cd5c208c3e9dc9640d854726f28a5b4161ae823\": container with ID starting with 5bda053531e2ef461d2df6060cd5c208c3e9dc9640d854726f28a5b4161ae823 not found: ID does not exist" Jul 10 17:29:16.950983 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.950912 2567 scope.go:117] "RemoveContainer" containerID="aa8a0a808f9c9837822db1176e33102621abb9590ab500938917ed0abae53c8c" Jul 10 17:29:16.951457 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:29:16.951422 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa8a0a808f9c9837822db1176e33102621abb9590ab500938917ed0abae53c8c\": container with ID starting with aa8a0a808f9c9837822db1176e33102621abb9590ab500938917ed0abae53c8c not found: ID does not exist" containerID="aa8a0a808f9c9837822db1176e33102621abb9590ab500938917ed0abae53c8c" Jul 10 17:29:16.951555 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.951458 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa8a0a808f9c9837822db1176e33102621abb9590ab500938917ed0abae53c8c"} err="failed to get container status \"aa8a0a808f9c9837822db1176e33102621abb9590ab500938917ed0abae53c8c\": rpc error: code = NotFound desc = could not find container \"aa8a0a808f9c9837822db1176e33102621abb9590ab500938917ed0abae53c8c\": container with ID starting with aa8a0a808f9c9837822db1176e33102621abb9590ab500938917ed0abae53c8c not found: ID does not exist" Jul 10 17:29:16.953366 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:16.953347 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1775b6-openshift-default-6d8b4f5447-8wsz5"] Jul 10 17:29:18.682177 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:18.682138 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3432fd86-c319-4f26-b5c8-d9a2920386dc" path="/var/lib/kubelet/pods/3432fd86-c319-4f26-b5c8-d9a2920386dc/volumes" Jul 10 17:29:19.087497 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.087466 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg"] Jul 10 17:29:19.087858 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.087847 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerName="istio-proxy" Jul 10 17:29:19.087899 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.087860 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerName="istio-proxy" Jul 10 17:29:19.087899 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.087869 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856eee9d-546f-42cc-9f3d-d0f46bd1ac14" containerName="maas-api" Jul 10 17:29:19.087899 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.087875 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="856eee9d-546f-42cc-9f3d-d0f46bd1ac14" containerName="maas-api" Jul 10 17:29:19.087899 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.087887 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6937ddba-5199-4c24-8f94-b47ae161f7ba" containerName="maas-api" Jul 10 17:29:19.087899 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.087893 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6937ddba-5199-4c24-8f94-b47ae161f7ba" containerName="maas-api" Jul 10 17:29:19.088045 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.087901 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerName="istio-proxy" Jul 10 17:29:19.088045 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.087908 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerName="istio-proxy" Jul 10 17:29:19.088045 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.087968 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerName="istio-proxy" Jul 10 17:29:19.088045 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.087977 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="856eee9d-546f-42cc-9f3d-d0f46bd1ac14" containerName="maas-api" Jul 10 17:29:19.088045 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.087987 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6937ddba-5199-4c24-8f94-b47ae161f7ba" containerName="maas-api" Jul 10 17:29:19.092462 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.092440 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.095043 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.095022 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-43091a-openshift-default-dockercfg-z5m8m\"" Jul 10 17:29:19.100921 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.100900 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg"] Jul 10 17:29:19.172621 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.172581 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-istio-envoy\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.172833 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.172643 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-workload-certs\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.172833 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.172671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196454d3-532e-44a4-a472-26981c68e0f2-istio-podinfo\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.172833 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.172699 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-credential-socket\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.172833 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.172721 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-istio-data\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.172833 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.172744 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hv4t\" (UniqueName: \"kubernetes.io/projected/196454d3-532e-44a4-a472-26981c68e0f2-kube-api-access-7hv4t\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.172833 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.172778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196454d3-532e-44a4-a472-26981c68e0f2-istiod-ca-cert\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.173070 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.172878 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196454d3-532e-44a4-a472-26981c68e0f2-istio-token\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.173070 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.172915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-workload-socket\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.274516 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.274468 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-workload-certs\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.274731 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.274524 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196454d3-532e-44a4-a472-26981c68e0f2-istio-podinfo\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.274731 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.274560 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-credential-socket\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.274731 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.274589 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-istio-data\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.274731 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.274614 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hv4t\" (UniqueName: \"kubernetes.io/projected/196454d3-532e-44a4-a472-26981c68e0f2-kube-api-access-7hv4t\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.274731 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.274651 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196454d3-532e-44a4-a472-26981c68e0f2-istiod-ca-cert\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.274994 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.274739 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196454d3-532e-44a4-a472-26981c68e0f2-istio-token\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.274994 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.274774 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-workload-socket\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.274994 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.274817 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-istio-envoy\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.274994 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.274952 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-workload-certs\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.275212 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.275013 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-credential-socket\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.275262 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.275215 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-workload-socket\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.275326 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.275300 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-istio-data\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.275448 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.275427 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196454d3-532e-44a4-a472-26981c68e0f2-istiod-ca-cert\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.277151 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.277132 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196454d3-532e-44a4-a472-26981c68e0f2-istio-podinfo\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.277572 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.277546 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-istio-envoy\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.290010 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.289981 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hv4t\" (UniqueName: \"kubernetes.io/projected/196454d3-532e-44a4-a472-26981c68e0f2-kube-api-access-7hv4t\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.295235 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.295208 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196454d3-532e-44a4-a472-26981c68e0f2-istio-token\") pod \"e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.404196 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.404106 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:19.553317 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.553265 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg"] Jul 10 17:29:19.555238 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:29:19.555212 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196454d3_532e_44a4_a472_26981c68e0f2.slice/crio-26ab4f2ab1a69b57ab0c665087c01a433d58e0b971b5e574f2aa12a65b416833 WatchSource:0}: Error finding container 26ab4f2ab1a69b57ab0c665087c01a433d58e0b971b5e574f2aa12a65b416833: Status 404 returned error can't find the container with id 26ab4f2ab1a69b57ab0c665087c01a433d58e0b971b5e574f2aa12a65b416833 Jul 10 17:29:19.557541 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.557506 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:29:19.557630 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.557586 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:29:19.557630 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.557617 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:29:19.949452 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.949411 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" event={"ID":"196454d3-532e-44a4-a472-26981c68e0f2","Type":"ContainerStarted","Data":"c79110b12d0e786b242f1f78e43124f115e9f644835d8533d168fe81083d46e9"} Jul 10 17:29:19.949842 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.949461 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" event={"ID":"196454d3-532e-44a4-a472-26981c68e0f2","Type":"ContainerStarted","Data":"26ab4f2ab1a69b57ab0c665087c01a433d58e0b971b5e574f2aa12a65b416833"} Jul 10 17:29:19.972756 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:19.972707 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" podStartSLOduration=0.97269086 podStartE2EDuration="972.69086ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:29:19.970582295 +0000 UTC m=+2089.857873615" watchObservedRunningTime="2026-07-10 17:29:19.97269086 +0000 UTC m=+2089.859982181" Jul 10 17:29:20.404946 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:20.404853 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:20.409723 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:20.409696 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:20.954967 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:20.954934 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:20.955942 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:20.955924 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:22.030099 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.030068 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp"] Jul 10 17:29:22.030752 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.030634 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3432fd86-c319-4f26-b5c8-d9a2920386dc" containerName="istio-proxy" Jul 10 17:29:22.033839 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.033823 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:29:22.036062 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.036040 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-43091a-serving-cert\"" Jul 10 17:29:22.043380 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.043351 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp"] Jul 10 17:29:22.104888 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.104850 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp\" (UID: \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:29:22.104888 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.104898 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndgst\" (UniqueName: \"kubernetes.io/projected/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-kube-api-access-ndgst\") pod \"maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp\" (UID: \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:29:22.105074 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.104969 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-maas-api-tls\") pod \"maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp\" (UID: \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:29:22.205988 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.205950 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp\" (UID: \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:29:22.205988 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.205988 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndgst\" (UniqueName: \"kubernetes.io/projected/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-kube-api-access-ndgst\") pod \"maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp\" (UID: \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:29:22.206247 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.206014 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-maas-api-tls\") pod \"maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp\" (UID: \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:29:22.206701 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.206662 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp\" (UID: \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:29:22.208624 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.208592 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-maas-api-tls\") pod \"maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp\" (UID: \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:29:22.215403 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.215379 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndgst\" (UniqueName: \"kubernetes.io/projected/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-kube-api-access-ndgst\") pod \"maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp\" (UID: \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:29:22.350331 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.350240 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:29:22.512033 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.512007 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp"] Jul 10 17:29:22.514864 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:29:22.514824 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db1a7c6_f8cb_4931_a453_bf98cdd8cf41.slice/crio-cdf2825c639b45b9509d3a33b6a3aa95ad5e9d45396b1c0ccf113600a6290d42 WatchSource:0}: Error finding container cdf2825c639b45b9509d3a33b6a3aa95ad5e9d45396b1c0ccf113600a6290d42: Status 404 returned error can't find the container with id cdf2825c639b45b9509d3a33b6a3aa95ad5e9d45396b1c0ccf113600a6290d42 Jul 10 17:29:22.967899 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:22.967863 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" event={"ID":"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41","Type":"ContainerStarted","Data":"cdf2825c639b45b9509d3a33b6a3aa95ad5e9d45396b1c0ccf113600a6290d42"} Jul 10 17:29:23.974341 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:23.974296 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" event={"ID":"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41","Type":"ContainerStarted","Data":"96548a73234d5a853ef0d91e46785f8d6efec3b9039cbcc1f3bddd52c22581ac"} Jul 10 17:29:23.974776 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:23.974422 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:29:23.989612 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:23.989551 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" podStartSLOduration=1.403279607 podStartE2EDuration="1.989532837s" totalImagesPullingTime="586.25323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:29:23.988928402 +0000 UTC m=+2093.876219747" watchObservedRunningTime="2026-07-10 17:29:23.989532837 +0000 UTC m=+2093.876824156" Jul 10 17:29:29.987254 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:29.987220 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:29:30.688369 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:30.688334 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:29:30.695418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:30.695395 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:29:30.824316 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:30.824265 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w"] Jul 10 17:29:30.828773 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:30.828740 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:30.832466 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:30.831233 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-2482f8-serving-cert\"" Jul 10 17:29:30.838838 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:30.838619 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w"] Jul 10 17:29:30.898356 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:30.895775 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxvlv\" (UniqueName: \"kubernetes.io/projected/d81aa4c3-f085-4875-a8f4-5659089e4756-kube-api-access-xxvlv\") pod \"maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w\" (UID: \"d81aa4c3-f085-4875-a8f4-5659089e4756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:30.898356 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:30.895944 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81aa4c3-f085-4875-a8f4-5659089e4756-maas-api-tls\") pod \"maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w\" (UID: \"d81aa4c3-f085-4875-a8f4-5659089e4756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:30.898356 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:30.895976 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81aa4c3-f085-4875-a8f4-5659089e4756-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w\" (UID: \"d81aa4c3-f085-4875-a8f4-5659089e4756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:30.997688 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:30.997636 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81aa4c3-f085-4875-a8f4-5659089e4756-maas-api-tls\") pod \"maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w\" (UID: \"d81aa4c3-f085-4875-a8f4-5659089e4756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:30.998246 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:30.997698 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81aa4c3-f085-4875-a8f4-5659089e4756-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w\" (UID: \"d81aa4c3-f085-4875-a8f4-5659089e4756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:30.998246 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:30.997746 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxvlv\" (UniqueName: \"kubernetes.io/projected/d81aa4c3-f085-4875-a8f4-5659089e4756-kube-api-access-xxvlv\") pod \"maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w\" (UID: \"d81aa4c3-f085-4875-a8f4-5659089e4756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:30.998246 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:29:30.998166 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-serving-cert: secret "maas-api-e2e-shared-b-2482f8-serving-cert" not found Jul 10 17:29:30.998246 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:29:30.998243 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81aa4c3-f085-4875-a8f4-5659089e4756-maas-api-tls podName:d81aa4c3-f085-4875-a8f4-5659089e4756 nodeName:}" failed. No retries permitted until 2026-07-10 17:29:31.498218694 +0000 UTC m=+2101.385509994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d81aa4c3-f085-4875-a8f4-5659089e4756-maas-api-tls") pod "maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" (UID: "d81aa4c3-f085-4875-a8f4-5659089e4756") : secret "maas-api-e2e-shared-b-2482f8-serving-cert" not found Jul 10 17:29:30.999239 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:30.999210 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81aa4c3-f085-4875-a8f4-5659089e4756-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w\" (UID: \"d81aa4c3-f085-4875-a8f4-5659089e4756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:31.009445 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:31.009416 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxvlv\" (UniqueName: \"kubernetes.io/projected/d81aa4c3-f085-4875-a8f4-5659089e4756-kube-api-access-xxvlv\") pod \"maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w\" (UID: \"d81aa4c3-f085-4875-a8f4-5659089e4756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:31.505217 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:31.505171 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81aa4c3-f085-4875-a8f4-5659089e4756-maas-api-tls\") pod \"maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w\" (UID: \"d81aa4c3-f085-4875-a8f4-5659089e4756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:31.507918 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:31.507882 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81aa4c3-f085-4875-a8f4-5659089e4756-maas-api-tls\") pod \"maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w\" (UID: \"d81aa4c3-f085-4875-a8f4-5659089e4756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:31.743859 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:31.743811 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:31.932581 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:31.932556 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w"] Jul 10 17:29:31.934457 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:29:31.934421 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81aa4c3_f085_4875_a8f4_5659089e4756.slice/crio-6764efcb4a46a58df5de1e4a218f6401993667a09f843e5f82a112861df4a175 WatchSource:0}: Error finding container 6764efcb4a46a58df5de1e4a218f6401993667a09f843e5f82a112861df4a175: Status 404 returned error can't find the container with id 6764efcb4a46a58df5de1e4a218f6401993667a09f843e5f82a112861df4a175 Jul 10 17:29:32.027337 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:32.027239 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" event={"ID":"d81aa4c3-f085-4875-a8f4-5659089e4756","Type":"ContainerStarted","Data":"6764efcb4a46a58df5de1e4a218f6401993667a09f843e5f82a112861df4a175"} Jul 10 17:29:33.034519 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:33.034485 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" event={"ID":"d81aa4c3-f085-4875-a8f4-5659089e4756","Type":"ContainerStarted","Data":"dca604b460ce83336182ae2968b8e4b673b5d6d2bb84b7df6cc49b41a776be73"} Jul 10 17:29:33.034949 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:33.034609 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:33.050199 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:33.050133 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" podStartSLOduration=2.717981912 podStartE2EDuration="3.050113468s" totalImagesPullingTime="332.131556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:29:33.049752175 +0000 UTC m=+2102.937043495" watchObservedRunningTime="2026-07-10 17:29:33.050113468 +0000 UTC m=+2102.937404793" Jul 10 17:29:36.996521 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:36.996460 2567 patch_prober.go:28] interesting pod/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.135.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 17:29:36.996905 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:36.996554 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" podUID="196454d3-532e-44a4-a472-26981c68e0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 17:29:38.063085 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:38.063052 2567 generic.go:356] "Generic (PLEG): container finished" podID="196454d3-532e-44a4-a472-26981c68e0f2" containerID="c79110b12d0e786b242f1f78e43124f115e9f644835d8533d168fe81083d46e9" exitCode=0 Jul 10 17:29:38.063548 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:38.063131 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" event={"ID":"196454d3-532e-44a4-a472-26981c68e0f2","Type":"ContainerDied","Data":"c79110b12d0e786b242f1f78e43124f115e9f644835d8533d168fe81083d46e9"} Jul 10 17:29:38.063648 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:38.063627 2567 scope.go:117] "RemoveContainer" containerID="c79110b12d0e786b242f1f78e43124f115e9f644835d8533d168fe81083d46e9" Jul 10 17:29:38.064716 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:38.064682 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:29:38.064823 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:38.064768 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:29:38.064823 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:38.064815 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:29:39.044669 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:39.044642 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:29:39.071525 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:39.071489 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" event={"ID":"196454d3-532e-44a4-a472-26981c68e0f2","Type":"ContainerStarted","Data":"65e789a1234e855bc0374be98044b21ca2866a2189c9cde4c41b23cbd80ef0e4"} Jul 10 17:29:39.404602 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:39.404498 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:39.404772 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:39.404697 2567 patch_prober.go:28] interesting pod/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.70:15021/healthz/ready\": dial tcp 10.135.0.70:15021: connect: connection refused" start-of-body= Jul 10 17:29:39.404824 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:39.404765 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" podUID="196454d3-532e-44a4-a472-26981c68e0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.70:15021/healthz/ready\": dial tcp 10.135.0.70:15021: connect: connection refused" Jul 10 17:29:40.405058 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:40.405017 2567 patch_prober.go:28] interesting pod/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.70:15021/healthz/ready\": dial tcp 10.135.0.70:15021: connect: connection refused" start-of-body= Jul 10 17:29:40.405475 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:40.405076 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" podUID="196454d3-532e-44a4-a472-26981c68e0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.70:15021/healthz/ready\": dial tcp 10.135.0.70:15021: connect: connection refused" Jul 10 17:29:41.405311 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:41.405246 2567 patch_prober.go:28] interesting pod/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.70:15021/healthz/ready\": dial tcp 10.135.0.70:15021: connect: connection refused" start-of-body= Jul 10 17:29:41.405793 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:41.405348 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" podUID="196454d3-532e-44a4-a472-26981c68e0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.70:15021/healthz/ready\": dial tcp 10.135.0.70:15021: connect: connection refused" Jul 10 17:29:42.405337 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:42.405301 2567 patch_prober.go:28] interesting pod/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.70:15021/healthz/ready\": dial tcp 10.135.0.70:15021: connect: connection refused" start-of-body= Jul 10 17:29:42.405742 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:42.405363 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" podUID="196454d3-532e-44a4-a472-26981c68e0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.70:15021/healthz/ready\": dial tcp 10.135.0.70:15021: connect: connection refused" Jul 10 17:29:43.404817 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:43.404775 2567 patch_prober.go:28] interesting pod/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.135.0.70:15021/healthz/ready\": dial tcp 10.135.0.70:15021: connect: connection refused" start-of-body= Jul 10 17:29:43.405006 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:43.404838 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" podUID="196454d3-532e-44a4-a472-26981c68e0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.135.0.70:15021/healthz/ready\": dial tcp 10.135.0.70:15021: connect: connection refused" Jul 10 17:29:44.408375 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:44.408346 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:44.408753 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:44.408609 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:29:44.409212 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:29:44.409193 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:34:30.717223 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:34:30.717182 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:34:30.725763 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:34:30.725743 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trj2g_da59a0ce-b509-4f6e-8685-b25d58a3fc2d/ovn-acl-logging/0.log" Jul 10 17:35:30.904224 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:35:30.904180 2567 scope.go:117] "RemoveContainer" containerID="9e1989de82a2be30412ed188380e703f397186cbefbebd972abc6b5977e90f45" Jul 10 17:36:02.937040 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:02.936996 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp"] Jul 10 17:36:02.937536 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:02.937293 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" podUID="7db1a7c6-f8cb-4931-a453-bf98cdd8cf41" containerName="maas-api" containerID="cri-o://96548a73234d5a853ef0d91e46785f8d6efec3b9039cbcc1f3bddd52c22581ac" gracePeriod=30 Jul 10 17:36:03.402775 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:03.402748 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:36:03.483198 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:03.483113 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndgst\" (UniqueName: \"kubernetes.io/projected/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-kube-api-access-ndgst\") pod \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\" (UID: \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\") " Jul 10 17:36:03.483198 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:03.483168 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-openshift-service-ca\") pod \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\" (UID: \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\") " Jul 10 17:36:03.483401 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:03.483257 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-maas-api-tls\") pod \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\" (UID: \"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41\") " Jul 10 17:36:03.483653 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:03.483619 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7db1a7c6-f8cb-4931-a453-bf98cdd8cf41" (UID: "7db1a7c6-f8cb-4931-a453-bf98cdd8cf41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:36:03.485467 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:03.485438 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7db1a7c6-f8cb-4931-a453-bf98cdd8cf41" (UID: "7db1a7c6-f8cb-4931-a453-bf98cdd8cf41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:36:03.485565 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:03.485509 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-kube-api-access-ndgst" (OuterVolumeSpecName: "kube-api-access-ndgst") pod "7db1a7c6-f8cb-4931-a453-bf98cdd8cf41" (UID: "7db1a7c6-f8cb-4931-a453-bf98cdd8cf41"). InnerVolumeSpecName "kube-api-access-ndgst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:36:03.584228 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:03.584176 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndgst\" (UniqueName: \"kubernetes.io/projected/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-kube-api-access-ndgst\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:03.584228 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:03.584227 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-openshift-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:03.584475 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:03.584243 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41-maas-api-tls\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:04.094697 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:04.094661 2567 generic.go:356] "Generic (PLEG): container finished" podID="7db1a7c6-f8cb-4931-a453-bf98cdd8cf41" containerID="96548a73234d5a853ef0d91e46785f8d6efec3b9039cbcc1f3bddd52c22581ac" exitCode=0 Jul 10 17:36:04.095170 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:04.094702 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" Jul 10 17:36:04.095170 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:04.094711 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" event={"ID":"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41","Type":"ContainerDied","Data":"96548a73234d5a853ef0d91e46785f8d6efec3b9039cbcc1f3bddd52c22581ac"} Jul 10 17:36:04.095170 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:04.094768 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp" event={"ID":"7db1a7c6-f8cb-4931-a453-bf98cdd8cf41","Type":"ContainerDied","Data":"cdf2825c639b45b9509d3a33b6a3aa95ad5e9d45396b1c0ccf113600a6290d42"} Jul 10 17:36:04.095170 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:04.094798 2567 scope.go:117] "RemoveContainer" containerID="96548a73234d5a853ef0d91e46785f8d6efec3b9039cbcc1f3bddd52c22581ac" Jul 10 17:36:04.104438 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:04.104419 2567 scope.go:117] "RemoveContainer" containerID="96548a73234d5a853ef0d91e46785f8d6efec3b9039cbcc1f3bddd52c22581ac" Jul 10 17:36:04.104727 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:36:04.104706 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96548a73234d5a853ef0d91e46785f8d6efec3b9039cbcc1f3bddd52c22581ac\": container with ID starting with 96548a73234d5a853ef0d91e46785f8d6efec3b9039cbcc1f3bddd52c22581ac not found: ID does not exist" containerID="96548a73234d5a853ef0d91e46785f8d6efec3b9039cbcc1f3bddd52c22581ac" Jul 10 17:36:04.104790 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:04.104735 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96548a73234d5a853ef0d91e46785f8d6efec3b9039cbcc1f3bddd52c22581ac"} err="failed to get container status \"96548a73234d5a853ef0d91e46785f8d6efec3b9039cbcc1f3bddd52c22581ac\": rpc error: code = NotFound desc = could not find container \"96548a73234d5a853ef0d91e46785f8d6efec3b9039cbcc1f3bddd52c22581ac\": container with ID starting with 96548a73234d5a853ef0d91e46785f8d6efec3b9039cbcc1f3bddd52c22581ac not found: ID does not exist" Jul 10 17:36:04.115477 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:04.115446 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp"] Jul 10 17:36:04.117296 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:04.117257 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-43091a-56ccd64c6c-hcdbp"] Jul 10 17:36:04.683340 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:04.683303 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7db1a7c6-f8cb-4931-a453-bf98cdd8cf41" path="/var/lib/kubelet/pods/7db1a7c6-f8cb-4931-a453-bf98cdd8cf41/volumes" Jul 10 17:36:08.846805 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:08.846773 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg"] Jul 10 17:36:08.847326 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:08.846990 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" podUID="196454d3-532e-44a4-a472-26981c68e0f2" containerName="istio-proxy" containerID="cri-o://65e789a1234e855bc0374be98044b21ca2866a2189c9cde4c41b23cbd80ef0e4" gracePeriod=30 Jul 10 17:36:10.074559 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:10.074517 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w"] Jul 10 17:36:10.075080 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:10.074766 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" podUID="d81aa4c3-f085-4875-a8f4-5659089e4756" containerName="maas-api" containerID="cri-o://dca604b460ce83336182ae2968b8e4b673b5d6d2bb84b7df6cc49b41a776be73" gracePeriod=30 Jul 10 17:36:10.334669 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:10.334595 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:36:10.452585 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:10.452546 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81aa4c3-f085-4875-a8f4-5659089e4756-maas-api-tls\") pod \"d81aa4c3-f085-4875-a8f4-5659089e4756\" (UID: \"d81aa4c3-f085-4875-a8f4-5659089e4756\") " Jul 10 17:36:10.452790 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:10.452670 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxvlv\" (UniqueName: \"kubernetes.io/projected/d81aa4c3-f085-4875-a8f4-5659089e4756-kube-api-access-xxvlv\") pod \"d81aa4c3-f085-4875-a8f4-5659089e4756\" (UID: \"d81aa4c3-f085-4875-a8f4-5659089e4756\") " Jul 10 17:36:10.452790 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:10.452735 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81aa4c3-f085-4875-a8f4-5659089e4756-openshift-service-ca\") pod \"d81aa4c3-f085-4875-a8f4-5659089e4756\" (UID: \"d81aa4c3-f085-4875-a8f4-5659089e4756\") " Jul 10 17:36:10.453079 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:10.453055 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81aa4c3-f085-4875-a8f4-5659089e4756-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d81aa4c3-f085-4875-a8f4-5659089e4756" (UID: "d81aa4c3-f085-4875-a8f4-5659089e4756"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:36:10.454881 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:10.454857 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81aa4c3-f085-4875-a8f4-5659089e4756-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d81aa4c3-f085-4875-a8f4-5659089e4756" (UID: "d81aa4c3-f085-4875-a8f4-5659089e4756"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:36:10.454965 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:10.454890 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81aa4c3-f085-4875-a8f4-5659089e4756-kube-api-access-xxvlv" (OuterVolumeSpecName: "kube-api-access-xxvlv") pod "d81aa4c3-f085-4875-a8f4-5659089e4756" (UID: "d81aa4c3-f085-4875-a8f4-5659089e4756"). InnerVolumeSpecName "kube-api-access-xxvlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:36:10.553660 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:10.553619 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81aa4c3-f085-4875-a8f4-5659089e4756-openshift-service-ca\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:10.553660 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:10.553651 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81aa4c3-f085-4875-a8f4-5659089e4756-maas-api-tls\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:10.553660 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:10.553664 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxvlv\" (UniqueName: \"kubernetes.io/projected/d81aa4c3-f085-4875-a8f4-5659089e4756-kube-api-access-xxvlv\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:11.132242 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:11.132145 2567 generic.go:356] "Generic (PLEG): container finished" podID="d81aa4c3-f085-4875-a8f4-5659089e4756" containerID="dca604b460ce83336182ae2968b8e4b673b5d6d2bb84b7df6cc49b41a776be73" exitCode=0 Jul 10 17:36:11.132242 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:11.132185 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" Jul 10 17:36:11.132242 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:11.132230 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" event={"ID":"d81aa4c3-f085-4875-a8f4-5659089e4756","Type":"ContainerDied","Data":"dca604b460ce83336182ae2968b8e4b673b5d6d2bb84b7df6cc49b41a776be73"} Jul 10 17:36:11.132794 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:11.132292 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w" event={"ID":"d81aa4c3-f085-4875-a8f4-5659089e4756","Type":"ContainerDied","Data":"6764efcb4a46a58df5de1e4a218f6401993667a09f843e5f82a112861df4a175"} Jul 10 17:36:11.132794 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:11.132315 2567 scope.go:117] "RemoveContainer" containerID="dca604b460ce83336182ae2968b8e4b673b5d6d2bb84b7df6cc49b41a776be73" Jul 10 17:36:11.141572 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:11.141539 2567 scope.go:117] "RemoveContainer" containerID="dca604b460ce83336182ae2968b8e4b673b5d6d2bb84b7df6cc49b41a776be73" Jul 10 17:36:11.141824 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:36:11.141805 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca604b460ce83336182ae2968b8e4b673b5d6d2bb84b7df6cc49b41a776be73\": container with ID starting with dca604b460ce83336182ae2968b8e4b673b5d6d2bb84b7df6cc49b41a776be73 not found: ID does not exist" containerID="dca604b460ce83336182ae2968b8e4b673b5d6d2bb84b7df6cc49b41a776be73" Jul 10 17:36:11.141866 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:11.141834 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca604b460ce83336182ae2968b8e4b673b5d6d2bb84b7df6cc49b41a776be73"} err="failed to get container status \"dca604b460ce83336182ae2968b8e4b673b5d6d2bb84b7df6cc49b41a776be73\": rpc error: code = NotFound desc = could not find container \"dca604b460ce83336182ae2968b8e4b673b5d6d2bb84b7df6cc49b41a776be73\": container with ID starting with dca604b460ce83336182ae2968b8e4b673b5d6d2bb84b7df6cc49b41a776be73 not found: ID does not exist" Jul 10 17:36:11.148324 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:11.148302 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w"] Jul 10 17:36:11.152456 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:11.152436 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2482f8-64b94bf456-dcp6w"] Jul 10 17:36:12.681821 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:12.681788 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d81aa4c3-f085-4875-a8f4-5659089e4756" path="/var/lib/kubelet/pods/d81aa4c3-f085-4875-a8f4-5659089e4756/volumes" Jul 10 17:36:14.105204 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.105180 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:36:14.149347 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.149215 2567 generic.go:356] "Generic (PLEG): container finished" podID="196454d3-532e-44a4-a472-26981c68e0f2" containerID="65e789a1234e855bc0374be98044b21ca2866a2189c9cde4c41b23cbd80ef0e4" exitCode=0 Jul 10 17:36:14.149347 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.149313 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" Jul 10 17:36:14.149566 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.149338 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" event={"ID":"196454d3-532e-44a4-a472-26981c68e0f2","Type":"ContainerDied","Data":"65e789a1234e855bc0374be98044b21ca2866a2189c9cde4c41b23cbd80ef0e4"} Jul 10 17:36:14.149566 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.149433 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg" event={"ID":"196454d3-532e-44a4-a472-26981c68e0f2","Type":"ContainerDied","Data":"26ab4f2ab1a69b57ab0c665087c01a433d58e0b971b5e574f2aa12a65b416833"} Jul 10 17:36:14.149566 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.149466 2567 scope.go:117] "RemoveContainer" containerID="65e789a1234e855bc0374be98044b21ca2866a2189c9cde4c41b23cbd80ef0e4" Jul 10 17:36:14.159102 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.159082 2567 scope.go:117] "RemoveContainer" containerID="c79110b12d0e786b242f1f78e43124f115e9f644835d8533d168fe81083d46e9" Jul 10 17:36:14.167217 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.167194 2567 scope.go:117] "RemoveContainer" containerID="65e789a1234e855bc0374be98044b21ca2866a2189c9cde4c41b23cbd80ef0e4" Jul 10 17:36:14.167493 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:36:14.167468 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e789a1234e855bc0374be98044b21ca2866a2189c9cde4c41b23cbd80ef0e4\": container with ID starting with 65e789a1234e855bc0374be98044b21ca2866a2189c9cde4c41b23cbd80ef0e4 not found: ID does not exist" containerID="65e789a1234e855bc0374be98044b21ca2866a2189c9cde4c41b23cbd80ef0e4" Jul 10 17:36:14.167566 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.167501 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e789a1234e855bc0374be98044b21ca2866a2189c9cde4c41b23cbd80ef0e4"} err="failed to get container status \"65e789a1234e855bc0374be98044b21ca2866a2189c9cde4c41b23cbd80ef0e4\": rpc error: code = NotFound desc = could not find container \"65e789a1234e855bc0374be98044b21ca2866a2189c9cde4c41b23cbd80ef0e4\": container with ID starting with 65e789a1234e855bc0374be98044b21ca2866a2189c9cde4c41b23cbd80ef0e4 not found: ID does not exist" Jul 10 17:36:14.167566 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.167518 2567 scope.go:117] "RemoveContainer" containerID="c79110b12d0e786b242f1f78e43124f115e9f644835d8533d168fe81083d46e9" Jul 10 17:36:14.167790 ip-10-0-131-207 kubenswrapper[2567]: E0710 17:36:14.167769 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79110b12d0e786b242f1f78e43124f115e9f644835d8533d168fe81083d46e9\": container with ID starting with c79110b12d0e786b242f1f78e43124f115e9f644835d8533d168fe81083d46e9 not found: ID does not exist" containerID="c79110b12d0e786b242f1f78e43124f115e9f644835d8533d168fe81083d46e9" Jul 10 17:36:14.167833 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.167797 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79110b12d0e786b242f1f78e43124f115e9f644835d8533d168fe81083d46e9"} err="failed to get container status \"c79110b12d0e786b242f1f78e43124f115e9f644835d8533d168fe81083d46e9\": rpc error: code = NotFound desc = could not find container \"c79110b12d0e786b242f1f78e43124f115e9f644835d8533d168fe81083d46e9\": container with ID starting with c79110b12d0e786b242f1f78e43124f115e9f644835d8533d168fe81083d46e9 not found: ID does not exist" Jul 10 17:36:14.290167 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.290125 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-credential-socket\") pod \"196454d3-532e-44a4-a472-26981c68e0f2\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " Jul 10 17:36:14.290418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.290185 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196454d3-532e-44a4-a472-26981c68e0f2-istiod-ca-cert\") pod \"196454d3-532e-44a4-a472-26981c68e0f2\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " Jul 10 17:36:14.290418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.290297 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196454d3-532e-44a4-a472-26981c68e0f2-istio-token\") pod \"196454d3-532e-44a4-a472-26981c68e0f2\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " Jul 10 17:36:14.290418 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.290390 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hv4t\" (UniqueName: \"kubernetes.io/projected/196454d3-532e-44a4-a472-26981c68e0f2-kube-api-access-7hv4t\") pod \"196454d3-532e-44a4-a472-26981c68e0f2\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " Jul 10 17:36:14.290609 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.290422 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196454d3-532e-44a4-a472-26981c68e0f2-istio-podinfo\") pod \"196454d3-532e-44a4-a472-26981c68e0f2\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " Jul 10 17:36:14.290609 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.290451 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-workload-certs\") pod \"196454d3-532e-44a4-a472-26981c68e0f2\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " Jul 10 17:36:14.290609 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.290517 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "196454d3-532e-44a4-a472-26981c68e0f2" (UID: "196454d3-532e-44a4-a472-26981c68e0f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:36:14.290735 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.290617 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-istio-envoy\") pod \"196454d3-532e-44a4-a472-26981c68e0f2\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " Jul 10 17:36:14.290735 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.290653 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196454d3-532e-44a4-a472-26981c68e0f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "196454d3-532e-44a4-a472-26981c68e0f2" (UID: "196454d3-532e-44a4-a472-26981c68e0f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:36:14.290735 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.290670 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "196454d3-532e-44a4-a472-26981c68e0f2" (UID: "196454d3-532e-44a4-a472-26981c68e0f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:36:14.290735 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.290711 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-workload-socket\") pod \"196454d3-532e-44a4-a472-26981c68e0f2\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " Jul 10 17:36:14.290935 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.290753 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-istio-data\") pod \"196454d3-532e-44a4-a472-26981c68e0f2\" (UID: \"196454d3-532e-44a4-a472-26981c68e0f2\") " Jul 10 17:36:14.290935 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.290922 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "196454d3-532e-44a4-a472-26981c68e0f2" (UID: "196454d3-532e-44a4-a472-26981c68e0f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:36:14.291342 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.291059 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-workload-socket\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:14.291342 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.291296 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-credential-socket\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:14.291342 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.291314 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196454d3-532e-44a4-a472-26981c68e0f2-istiod-ca-cert\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:14.291342 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.291329 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-workload-certs\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:14.292152 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.292124 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "196454d3-532e-44a4-a472-26981c68e0f2" (UID: "196454d3-532e-44a4-a472-26981c68e0f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:36:14.292665 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.292634 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/196454d3-532e-44a4-a472-26981c68e0f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "196454d3-532e-44a4-a472-26981c68e0f2" (UID: "196454d3-532e-44a4-a472-26981c68e0f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:36:14.292763 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.292696 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196454d3-532e-44a4-a472-26981c68e0f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "196454d3-532e-44a4-a472-26981c68e0f2" (UID: "196454d3-532e-44a4-a472-26981c68e0f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:36:14.292817 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.292775 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196454d3-532e-44a4-a472-26981c68e0f2-kube-api-access-7hv4t" (OuterVolumeSpecName: "kube-api-access-7hv4t") pod "196454d3-532e-44a4-a472-26981c68e0f2" (UID: "196454d3-532e-44a4-a472-26981c68e0f2"). InnerVolumeSpecName "kube-api-access-7hv4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:36:14.293128 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.293105 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "196454d3-532e-44a4-a472-26981c68e0f2" (UID: "196454d3-532e-44a4-a472-26981c68e0f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:36:14.392537 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.392506 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196454d3-532e-44a4-a472-26981c68e0f2-istio-token\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:14.392537 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.392535 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hv4t\" (UniqueName: \"kubernetes.io/projected/196454d3-532e-44a4-a472-26981c68e0f2-kube-api-access-7hv4t\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:14.392537 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.392546 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196454d3-532e-44a4-a472-26981c68e0f2-istio-podinfo\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:14.392771 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.392554 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-istio-envoy\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:14.392771 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.392562 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196454d3-532e-44a4-a472-26981c68e0f2-istio-data\") on node \"ip-10-0-131-207.ec2.internal\" DevicePath \"\"" Jul 10 17:36:14.471191 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.471160 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg"] Jul 10 17:36:14.475239 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.475212 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-43091a-openshift-default-7bb4ffc49f-w27mg"] Jul 10 17:36:14.684408 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:14.684372 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="196454d3-532e-44a4-a472-26981c68e0f2" path="/var/lib/kubelet/pods/196454d3-532e-44a4-a472-26981c68e0f2/volumes" Jul 10 17:36:16.536519 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:16.536487 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c4668c64b-7tjvg_7107d3ea-1665-470b-8424-c8c9734d1ae3/authorino/0.log" Jul 10 17:36:21.450533 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:21.450461 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-j98wg_12a523b0-50b9-4449-b3be-d66f9cf856de/manager/0.log" Jul 10 17:36:21.692223 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:21.692189 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c76c8d8b8-b2464_3a2202f3-4809-4b88-8837-0614397cb2dc/manager/0.log" Jul 10 17:36:21.811296 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:21.811197 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-flc2x_13085247-748f-408f-b5b1-504973bfd245/server/0.log" Jul 10 17:36:21.930482 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:21.930451 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-l2csp_2276b013-5fbb-4763-b462-1cd63e49e432/manager/0.log" Jul 10 17:36:22.168707 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:22.168628 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67764b8b78-jdvj6_007e50fc-7ce7-4946-89b9-9dcbb3d0c18a/copy-manifests/0.log" Jul 10 17:36:22.177683 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:22.177658 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67764b8b78-jdvj6_007e50fc-7ce7-4946-89b9-9dcbb3d0c18a/manager/0.log" Jul 10 17:36:22.669209 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:22.669178 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7f7754899c-6glkr_c135b866-8f99-4008-b9fe-3fb5576c1da8/maas-api/0.log" Jul 10 17:36:22.789644 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:22.789606 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-xh44p_160440f6-fcf4-4837-a170-cc0f4089df6c/postgres/0.log" Jul 10 17:36:24.120172 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:24.120136 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c4668c64b-7tjvg_7107d3ea-1665-470b-8424-c8c9734d1ae3/authorino/0.log" Jul 10 17:36:24.242761 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:24.242731 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-nx4cp_670c3697-d489-43a4-a30e-1e166b2cff98/manager/0.log" Jul 10 17:36:24.356662 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:24.356630 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fq7z9_7b8c8bfd-4bd4-4988-a44c-602b425174f0/manager/0.log" Jul 10 17:36:24.471092 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:24.471061 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pplmf_d4964467-0c88-43f5-8fb3-ac8fb7fba9bc/kuadrant-console-plugin/0.log" Jul 10 17:36:24.591114 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:24.591085 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-k7dz7_e193913a-dd57-464b-9e4d-b5d3e22af580/registry-server/0.log" Jul 10 17:36:24.720103 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:24.720074 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vzr5p_83610753-2dc9-448f-8763-becc2bc23acf/manager/0.log" Jul 10 17:36:25.564564 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:25.564531 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-558b5454d6-85xfq_3b5a1938-c8af-431e-be5c-6d4bb9f4a171/kube-auth-proxy/0.log" Jul 10 17:36:25.804089 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:25.804040 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7rph_f43b2410-1f24-432b-9437-b7010fe00be2/istio-proxy/1.log" Jul 10 17:36:25.923194 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:25.923120 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-pdwpf_de535c35-7176-4946-af88-727d337e2269/payload-pre-processing/0.log" Jul 10 17:36:26.041036 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:26.041004 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-2c5fz_d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f/payload-processing/0.log" Jul 10 17:36:26.528970 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:26.528935 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-t6xqn_8f3a27e0-516b-41eb-98aa-ba02855ffda2/main/0.log" Jul 10 17:36:26.651946 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:26.651914 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-6fgst_ecafcbdd-d66f-4652-aead-ed9b9cd07a62/main/0.log" Jul 10 17:36:26.892103 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:26.892017 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbk2kbw_1685e004-b215-4793-b784-c477c2df1823/main/0.log" Jul 10 17:36:27.127291 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:27.127234 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-cmsgp_747ba32a-9f68-4893-843a-5f2798a893f1/main/0.log" Jul 10 17:36:34.078876 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:34.078844 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rf7b9_e3390d32-503c-4e8f-ab8d-c1b3881f2df9/global-pull-secret-syncer/0.log" Jul 10 17:36:34.198864 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:34.198835 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n8qpc_441cb5b6-d258-4ccc-a950-62b1179f6c1a/konnectivity-agent/0.log" Jul 10 17:36:34.245200 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:34.245171 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-207.ec2.internal_b203731d72488e0ccf24b1f1be91c047/haproxy/0.log" Jul 10 17:36:39.476807 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:39.476777 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c4668c64b-7tjvg_7107d3ea-1665-470b-8424-c8c9734d1ae3/authorino/0.log" Jul 10 17:36:39.505218 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:39.505194 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-nx4cp_670c3697-d489-43a4-a30e-1e166b2cff98/manager/0.log" Jul 10 17:36:39.534553 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:39.534529 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fq7z9_7b8c8bfd-4bd4-4988-a44c-602b425174f0/manager/0.log" Jul 10 17:36:39.580058 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:39.580030 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pplmf_d4964467-0c88-43f5-8fb3-ac8fb7fba9bc/kuadrant-console-plugin/0.log" Jul 10 17:36:39.618752 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:39.618720 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-k7dz7_e193913a-dd57-464b-9e4d-b5d3e22af580/registry-server/0.log" Jul 10 17:36:39.744673 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:39.744588 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vzr5p_83610753-2dc9-448f-8763-becc2bc23acf/manager/0.log" Jul 10 17:36:41.856804 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:41.856773 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b4d5c6978-hr8v5_06a4fc65-bef5-44e1-b862-48d154fb2fc4/metrics-server/0.log" Jul 10 17:36:42.102599 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:42.102566 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4mpd_ba1d7c90-8728-431c-9e58-926cc6700a1a/node-exporter/0.log" Jul 10 17:36:42.125103 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:42.125030 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4mpd_ba1d7c90-8728-431c-9e58-926cc6700a1a/kube-rbac-proxy/0.log" Jul 10 17:36:42.149049 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:42.149029 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4mpd_ba1d7c90-8728-431c-9e58-926cc6700a1a/init-textfile/0.log" Jul 10 17:36:43.907113 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:43.907075 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-j4xzk_473995a2-4644-48b2-bb05-cc5aadfcc990/networking-console-plugin/0.log" Jul 10 17:36:44.674419 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.674384 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2"] Jul 10 17:36:44.674776 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.674764 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="196454d3-532e-44a4-a472-26981c68e0f2" containerName="istio-proxy" Jul 10 17:36:44.674822 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.674779 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="196454d3-532e-44a4-a472-26981c68e0f2" containerName="istio-proxy" Jul 10 17:36:44.674822 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.674794 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81aa4c3-f085-4875-a8f4-5659089e4756" containerName="maas-api" Jul 10 17:36:44.674822 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.674799 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81aa4c3-f085-4875-a8f4-5659089e4756" containerName="maas-api" Jul 10 17:36:44.674822 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.674808 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7db1a7c6-f8cb-4931-a453-bf98cdd8cf41" containerName="maas-api" Jul 10 17:36:44.674822 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.674814 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7db1a7c6-f8cb-4931-a453-bf98cdd8cf41" containerName="maas-api" Jul 10 17:36:44.674999 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.674874 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81aa4c3-f085-4875-a8f4-5659089e4756" containerName="maas-api" Jul 10 17:36:44.674999 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.674882 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="196454d3-532e-44a4-a472-26981c68e0f2" containerName="istio-proxy" Jul 10 17:36:44.674999 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.674889 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7db1a7c6-f8cb-4931-a453-bf98cdd8cf41" containerName="maas-api" Jul 10 17:36:44.678108 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.678085 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.680587 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.680563 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dnfk7\"/\"openshift-service-ca.crt\"" Jul 10 17:36:44.680733 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.680566 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dnfk7\"/\"kube-root-ca.crt\"" Jul 10 17:36:44.680733 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.680606 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dnfk7\"/\"default-dockercfg-gdhhc\"" Jul 10 17:36:44.689163 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.689143 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2"] Jul 10 17:36:44.777769 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.777734 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-sys\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.777769 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.777772 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-lib-modules\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.777967 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.777798 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qrsf\" (UniqueName: \"kubernetes.io/projected/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-kube-api-access-2qrsf\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.777967 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.777864 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-podres\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.777967 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.777893 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-proc\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.878550 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.878511 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-sys\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.878731 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.878555 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-lib-modules\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.878731 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.878645 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-sys\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.878731 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.878688 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qrsf\" (UniqueName: \"kubernetes.io/projected/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-kube-api-access-2qrsf\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.878731 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.878699 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-lib-modules\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.878896 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.878764 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-podres\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.878896 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.878795 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-proc\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.878896 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.878869 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-proc\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.878990 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.878923 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-podres\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.888246 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.888224 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qrsf\" (UniqueName: \"kubernetes.io/projected/9e7646e5-2a7d-48a7-9873-b2bde6d6fd89-kube-api-access-2qrsf\") pod \"perf-node-gather-daemonset-89kz2\" (UID: \"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:44.989352 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:44.989327 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:45.009517 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:45.009491 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-f6d755568-8gvcf_427e095c-ad94-495d-b6d3-aa6ea806257a/console/0.log" Jul 10 17:36:45.130406 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:45.130380 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2"] Jul 10 17:36:45.131996 ip-10-0-131-207 kubenswrapper[2567]: W0710 17:36:45.131960 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9e7646e5_2a7d_48a7_9873_b2bde6d6fd89.slice/crio-7d8357a5c80a87692acb1fb09d97cd4b328ef7e3b93d4a65323ce107bf7a6c32 WatchSource:0}: Error finding container 7d8357a5c80a87692acb1fb09d97cd4b328ef7e3b93d4a65323ce107bf7a6c32: Status 404 returned error can't find the container with id 7d8357a5c80a87692acb1fb09d97cd4b328ef7e3b93d4a65323ce107bf7a6c32 Jul 10 17:36:45.306861 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:45.306763 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" event={"ID":"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89","Type":"ContainerStarted","Data":"d07acac09a8655727cb32dc2a9663c45c9f3ebdd252044eec668a3f032ed3fcc"} Jul 10 17:36:45.306861 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:45.306834 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" event={"ID":"9e7646e5-2a7d-48a7-9873-b2bde6d6fd89","Type":"ContainerStarted","Data":"7d8357a5c80a87692acb1fb09d97cd4b328ef7e3b93d4a65323ce107bf7a6c32"} Jul 10 17:36:45.306861 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:45.306858 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:45.322792 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:45.322743 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" podStartSLOduration=1.32272744 podStartE2EDuration="1.32272744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:36:45.321231689 +0000 UTC m=+2535.208523010" watchObservedRunningTime="2026-07-10 17:36:45.32272744 +0000 UTC m=+2535.210018760" Jul 10 17:36:46.753356 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:46.753327 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m5jwp_f1c389de-199e-468c-851d-1a5d8a9e51fe/dns/0.log" Jul 10 17:36:46.773885 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:46.773862 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m5jwp_f1c389de-199e-468c-851d-1a5d8a9e51fe/kube-rbac-proxy/0.log" Jul 10 17:36:46.820874 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:46.820853 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kzbjz_dd8e4f0b-6e72-4a6e-bbb7-f7e33bc6cfff/dns-node-resolver/0.log" Jul 10 17:36:47.447314 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:47.447283 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g924d_5ac53225-b3f6-4a01-a823-03c876fdc239/node-ca/0.log" Jul 10 17:36:48.541788 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:48.541754 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-558b5454d6-85xfq_3b5a1938-c8af-431e-be5c-6d4bb9f4a171/kube-auth-proxy/0.log" Jul 10 17:36:48.594419 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:48.594392 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7rph_f43b2410-1f24-432b-9437-b7010fe00be2/istio-proxy/0.log" Jul 10 17:36:48.621868 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:48.621841 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7rph_f43b2410-1f24-432b-9437-b7010fe00be2/istio-proxy/1.log" Jul 10 17:36:48.649761 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:48.649738 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-pdwpf_de535c35-7176-4946-af88-727d337e2269/payload-pre-processing/0.log" Jul 10 17:36:48.679899 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:48.679874 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-2c5fz_d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f/payload-processing/0.log" Jul 10 17:36:49.436358 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:49.436328 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c64cd_853903d4-a52c-4bc3-a1ae-20a096602158/serve-healthcheck-canary/0.log" Jul 10 17:36:50.019303 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:50.019250 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n4jsc_793fd3dd-65e8-4ebf-87b1-8d5a9667b139/kube-rbac-proxy/0.log" Jul 10 17:36:50.039676 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:50.039649 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n4jsc_793fd3dd-65e8-4ebf-87b1-8d5a9667b139/exporter/0.log" Jul 10 17:36:50.061364 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:50.061339 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n4jsc_793fd3dd-65e8-4ebf-87b1-8d5a9667b139/extractor/0.log" Jul 10 17:36:51.323151 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:51.323122 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-89kz2" Jul 10 17:36:52.219304 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:52.219253 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-j98wg_12a523b0-50b9-4449-b3be-d66f9cf856de/manager/0.log" Jul 10 17:36:52.501376 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:52.501292 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c76c8d8b8-b2464_3a2202f3-4809-4b88-8837-0614397cb2dc/manager/0.log" Jul 10 17:36:52.521568 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:52.521536 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-flc2x_13085247-748f-408f-b5b1-504973bfd245/server/0.log" Jul 10 17:36:52.583551 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:52.583523 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-l2csp_2276b013-5fbb-4763-b462-1cd63e49e432/manager/0.log" Jul 10 17:36:52.653770 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:52.653740 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67764b8b78-jdvj6_007e50fc-7ce7-4946-89b9-9dcbb3d0c18a/manager/0.log" Jul 10 17:36:52.673581 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:52.673553 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67764b8b78-jdvj6_007e50fc-7ce7-4946-89b9-9dcbb3d0c18a/copy-manifests/0.log" Jul 10 17:36:55.236231 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:55.236203 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-558b5454d6-85xfq_3b5a1938-c8af-431e-be5c-6d4bb9f4a171/kube-auth-proxy/0.log" Jul 10 17:36:55.268050 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:55.268022 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7rph_f43b2410-1f24-432b-9437-b7010fe00be2/istio-proxy/0.log" Jul 10 17:36:55.297109 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:55.297086 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7rph_f43b2410-1f24-432b-9437-b7010fe00be2/istio-proxy/1.log" Jul 10 17:36:55.315726 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:55.315701 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-pdwpf_de535c35-7176-4946-af88-727d337e2269/payload-pre-processing/0.log" Jul 10 17:36:55.334374 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:55.334342 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-2c5fz_d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f/payload-processing/0.log" Jul 10 17:36:56.763421 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:56.763391 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-558b5454d6-85xfq_3b5a1938-c8af-431e-be5c-6d4bb9f4a171/kube-auth-proxy/0.log" Jul 10 17:36:56.798243 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:56.798214 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7rph_f43b2410-1f24-432b-9437-b7010fe00be2/istio-proxy/0.log" Jul 10 17:36:56.828353 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:56.828317 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7rph_f43b2410-1f24-432b-9437-b7010fe00be2/istio-proxy/1.log" Jul 10 17:36:56.851395 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:56.851360 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-pdwpf_de535c35-7176-4946-af88-727d337e2269/payload-pre-processing/0.log" Jul 10 17:36:56.873645 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:36:56.873608 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-2c5fz_d700cc88-6d5d-4fb5-8c73-c5869b2bbd4f/payload-processing/0.log" Jul 10 17:37:00.709903 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:37:00.709870 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4blh9_b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc/kube-multus-additional-cni-plugins/0.log" Jul 10 17:37:00.733769 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:37:00.733743 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4blh9_b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc/egress-router-binary-copy/0.log" Jul 10 17:37:00.760170 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:37:00.760147 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4blh9_b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc/cni-plugins/0.log" Jul 10 17:37:00.781761 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:37:00.781743 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4blh9_b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc/bond-cni-plugin/0.log" Jul 10 17:37:00.802183 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:37:00.802167 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4blh9_b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc/routeoverride-cni/0.log" Jul 10 17:37:00.822408 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:37:00.822384 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4blh9_b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc/whereabouts-cni-bincopy/0.log" Jul 10 17:37:00.842099 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:37:00.842068 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4blh9_b1a8d991-fc91-4ac2-9b25-66d8ff3eedcc/whereabouts-cni/0.log" Jul 10 17:37:01.180511 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:37:01.180469 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-d62w2_403bed03-bca0-461d-9cbf-767071e7a527/kube-multus/0.log" Jul 10 17:37:01.236898 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:37:01.236839 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6cldf_f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c/network-metrics-daemon/0.log" Jul 10 17:37:01.254326 ip-10-0-131-207 kubenswrapper[2567]: I0710 17:37:01.254303 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6cldf_f9a7cd3c-d5fa-4909-9a2e-8f8eaaab4f6c/kube-rbac-proxy/0.log"